Why does overparameterization in neural networks still work despite linear algebra suggesting redundancy?

Overparameterization increases solution space, making optimization easier rather than harder.

Linear algebra suggests redundant parameters reduce rank efficiency, but in deep learning, overparameterization creates a high-dimensional manifold of solutions where gradient descent can find low-loss regions more easily. Many solutions exist, and optimization benefits from smoother loss geometry and better-conditioned gradients.