What is catastrophic overparameterization in deep learning?

Overparameterization occurs when models have far more parameters than training samples.

Surprisingly, large networks generalize well due to implicit regularization of SGD, despite ability to perfectly fit training data.