seniorGradient Descent
What is stochastic noise in Gradient Descent?
Updated May 16, 2026
Short answer
Stochastic noise is randomness introduced by using subsets of data in gradient estimation.
Deep explanation
In Stochastic Gradient Descent, gradients are computed using mini-batches instead of full datasets, introducing variance in updates. This noise can help escape local minima but may slow convergence.
Real-world example
Training neural networks using mini-batches in TensorFlow or PyTorch.
Common mistakes
- Assuming noise is harmful
- it actually improves generalization.
Follow-up questions
- Why does noise help optimization?
- How to reduce noise?