seniorGradient Descent
What is the role of Hessian in optimization landscapes?
Updated May 16, 2026
Short answer
The Hessian describes curvature of the loss function in all parameter directions.
Deep explanation
The Hessian matrix contains second-order partial derivatives and reveals whether a point is a minimum, maximum, or saddle point. Positive definiteness indicates convex curvature, while negative eigenvalues indicate saddle directions. It is critical for Newton-based optimization methods.
Real-world example
Analyzing stability of deep learning minima.
Common mistakes
- Assuming gradient alone is sufficient for curvature understanding.
Follow-up questions
- What does eigenvalue sign indicate?
- Why is Hessian expensive?