How does Q-Learning handle function approximation errors and why can they compound?

Function approximation errors in Q-Learning can compound due to bootstrapping, leading to divergence or unstable policies.

When Q-Learning uses function approximators like neural networks, the estimated Q-values are not exact. Since updates rely on these estimates (bootstrapping), any error in Q(s', a') propagates backward into Q(s, a). Over repeated updates, these errors can amplify, especially when combined with off-policy learning and non-linear function approximators. This is one reason deep Q-learning requires stabilization techniques like target networks, replay buffers, and careful optimization.