seniorGradient Descent
What is gradient descent with momentum instability?
Updated May 16, 2026
Short answer
Momentum instability occurs when accumulated velocity overshoots minima.
Deep explanation
If momentum coefficient is too high or learning rate is large, the velocity term accumulates excessively, causing oscillations or divergence. This is common in sharp curvature regions where gradients change direction rapidly.
Real-world example
Training deep networks oscillating around minima during early experiments.
Common mistakes
- Setting momentum close to 1 without tuning learning rate.
Follow-up questions
- How to fix instability?
- What is safe momentum range?