seniorGradient Descent
What is Hessian-free optimization?
Updated May 16, 2026
Short answer
Hessian-free optimization approximates second-order methods without explicitly computing the Hessian.
Deep explanation
Instead of computing or storing the Hessian matrix, Hessian-free methods use iterative approximations like conjugate gradient to compute Hessian-vector products. This makes second-order optimization feasible for large neural networks.
Real-world example
Early deep learning research on recurrent neural networks.
Common mistakes
- Thinking full Hessian is explicitly computed.
Follow-up questions
- What is Hessian-vector product?
- Why is it scalable?