seniorGradient Descent
What is curvature explosion in Gradient Descent?
Updated May 16, 2026
Short answer
Curvature explosion occurs when second-order derivatives become extremely large, destabilizing updates.
Deep explanation
In regions of high curvature, small changes in parameters lead to large gradient changes, causing instability in Gradient Descent. This often leads to divergence unless learning rates are reduced or normalization is applied.
Real-world example
Training deep networks with poorly scaled activations.
Common mistakes
- Ignoring curvature effects and only tuning learning rate.
Follow-up questions
- How to detect it?
- How to fix it?