What causes training instability in very large language models?

Instability arises from gradient explosion, poor initialization, and optimization dynamics at scale.

As model size increases, gradients become noisy and sensitive to learning rate selection. Layer normalization, residual connections, and careful initialization mitigate instability. Mixed precision training introduces numerical instability if not properly scaled.