How does large-scale unsupervised optimization avoid representation collapse?

It uses normalization, contrastive objectives, and redundancy reduction techniques to maintain diversity in embeddings.

Representation collapse occurs when models map all inputs to identical embeddings. Large-scale systems prevent this using techniques like batch normalization, layer normalization, stop-gradient mechanisms (BYOL), variance regularization (VICReg), and temperature-scaled contrastive loss. These methods enforce variance and decorrelation constraints, ensuring embeddings remain informative.