seniorGradient Descent
What is signal-to-noise ratio (SNR) in SGD updates?
Updated May 16, 2026
Short answer
SNR measures how much useful gradient signal exists relative to stochastic noise.
Deep explanation
In SGD, each mini-batch gradient contains true signal plus noise. The signal-to-noise ratio determines optimization efficiency. Low SNR leads to slow convergence, while high SNR resembles batch gradient descent. SNR affects optimal batch size and learning rate scaling.
Real-world example
Large batch training in distributed deep learning systems.
Common mistakes
- Assuming larger batch size always improves training.
Follow-up questions
- What improves SNR?
- What reduces SNR?