How does gradient noise scale impact large model training stability?

Gradient noise increases with batch size and model size imbalance, affecting convergence stability.

Gradient noise scale (GNS) measures stochasticity in gradient estimates. Large batch training can lead to sharp minima or instability. Proper scaling of learning rates and adaptive optimizers are required to maintain stable convergence dynamics in LLM training.