seniorGradient Descent
What is Polyak step size rule?
Updated May 16, 2026
Short answer
Polyak step size adapts learning rate using current loss and optimal loss estimate.
Deep explanation
Instead of fixed learning rates, Polyak step size adjusts updates based on the gap between current loss and optimal loss. This makes convergence more adaptive and theoretically efficient when the optimal value is known or estimated.
Real-world example
Used in convex optimization problems with known lower bounds.
Common mistakes
- Assuming optimal loss is always known in practice.
Follow-up questions
- What is loss_star?
- When does it fail?