What is reinforcement learning instability in large language models?

RL instability occurs when policy updates diverge due to high variance rewards and unstable gradients.

In RLHF, reward signals are noisy and sparse, leading to unstable policy gradients. PPO mitigates this using clipping, but instability can still arise from reward model bias or distribution shift. Techniques like KL regularization and reward normalization are essential.