Why does gradient-based optimization depend on linear algebra structure of the loss landscape?

Because gradients, curvature, and updates are all defined in vector spaces.

The loss landscape in ML is a high-dimensional vector space where gradients represent directional derivatives and Hessians represent curvature. Optimization algorithms like SGD, Adam, and Newton’s method are fundamentally linear algebra operations on these structures. Convergence behavior depends on eigenvalues of the Hessian matrix.