seniorGradient Descent
What is gradient noise scale?
Updated May 16, 2026
Short answer
Gradient noise scale measures variability of stochastic gradients.
Deep explanation
It quantifies how noisy gradient estimates are compared to full-batch gradients. It helps determine optimal batch size and learning rate for efficient training.
Real-world example
Choosing batch size in training deep neural networks.
Common mistakes
- Ignoring noise scale when scaling training systems.
Follow-up questions
- Why is it important?
- What affects noise scale?