What is the impact of correlated samples in Q-Learning training?

Correlated samples cause unstable updates and slow convergence in Q-Learning.

Sequential environment transitions are highly correlated, violating the i.i.d. assumption required for stable gradient-based learning. This leads to oscillations and divergence in Q-value estimates. Experience replay mitigates this by randomly sampling past transitions, breaking temporal correlations and stabilizing learning.