Why does gradient descent behave differently in high dimensions?

Loss landscapes become anisotropic and poorly conditioned.

High-dimensional optimization surfaces have ravines and plateaus due to ill-conditioned Hessians, making convergence slow and sensitive to learning rates.