How does Q-Learning behave under function approximation + off-policy mismatch?

When combining function approximation with off-policy learning, Q-Learning can become unstable due to distribution mismatch and bootstrapped error amplification.

Q-Learning is inherently off-policy, meaning it learns from transitions generated by a behavior policy different from the greedy target policy. When combined with function approximation (like neural networks), this creates a mismatch between the data distribution and the policy being evaluated. The network is trained on states that may not be representative of the optimal policy’s visitation distribution, leading to extrapolation errors.…