Why does stochastic gradient descent implicitly perform matrix factorization?

SGD explores low-rank solutions in parameter space.

SGD dynamics bias learning toward low-complexity solutions. In matrix parameterizations, this often corresponds to implicit factorization where weight updates approximate low-rank structure over time.