What is the effect of high variance updates in Q-Learning training dynamics?

High variance updates destabilize learning by causing oscillations and inconsistent Q-value estimates.

In Q-Learning, variance arises from stochastic transitions, noisy rewards, and bootstrapped targets. High variance leads to unstable gradients, making Q-values oscillate or diverge. This is especially problematic when combined with function approximation. Techniques such as reward normalization, larger batch sizes, experience replay, and target networks help reduce variance and stabilize learning.