seniorGradient Descent
What is the Robbins-Monro algorithm and how does it relate to Gradient Descent?
Updated May 16, 2026
Short answer
Robbins-Monro is a foundational stochastic approximation method that underpins SGD.
Deep explanation
The Robbins-Monro algorithm solves root-finding problems when only noisy observations are available. It forms the theoretical basis for Stochastic Gradient Descent by showing that iterative updates with diminishing step sizes can converge despite noise. The key idea is that expectation of noisy gradients approximates true gradients over time, enabling convergence under certain step-size conditions.
Real-world example
Online ad systems adjusting click-through rate models from streaming noisy user data.
Common mistakes
- Using constant learning rates without satisfying convergence conditions.
Follow-up questions
- What are step size conditions for convergence?
- How does SGD violate or satisfy this?