seniorGradient Descent
What is curvature-driven optimization instability?
Updated May 16, 2026
Short answer
Instability occurs when steep curvature causes overshooting during updates.
Deep explanation
In regions where the Hessian has large eigenvalues, Gradient Descent updates can overshoot minima, causing oscillations or divergence. This is especially problematic in poorly conditioned optimization problems.
Real-world example
Training deep networks with poorly scaled features.
Common mistakes
- Using same learning rate for all parameters.
Follow-up questions
- What is eigenvalue role?
- How to fix instability?