What is policy collapse in Q-Learning and how does it occur?

Policy collapse happens when the agent converges prematurely to a suboptimal deterministic policy.

Due to insufficient exploration or biased Q-estimates, the agent may stop exploring alternative actions early. Once a suboptimal action appears best, epsilon-greedy exploitation reinforces it further, leading to collapse. This is common in sparse reward or high-noise environments.