What is catastrophic scaling instability in LLM training?

Scaling instability refers to divergence or degraded performance when increasing model size or batch size.

As models scale, optimization becomes unstable due to gradient noise, learning rate sensitivity, and normalization issues. Techniques like learning rate warmup, gradient clipping, and adaptive optimizers are critical.