seniorGradient Descent
What is preconditioning in optimization?
Updated May 16, 2026
Short answer
Preconditioning transforms gradients to improve convergence speed.
Deep explanation
Preconditioning rescales gradients using a transformation matrix that compensates for poor curvature. It effectively reshapes the loss surface into a more isotropic form, improving Gradient Descent efficiency.
Real-world example
Used in large-scale linear regression and scientific computing.
Common mistakes
- Applying preconditioning without understanding scaling effects.
Follow-up questions
- What is diagonal preconditioning?
- How is it related to Adam?