seniorLinear Algebra
What causes loss of rank in neural network representations?
Updated May 16, 2026
Short answer
Rank collapse happens when representations become linearly dependent.
Deep explanation
In deep networks, repeated linear transformations and normalization can cause embeddings to collapse into lower-dimensional subspaces. This reduces expressiveness and harms generalization. It often appears in deep representation learning and self-supervised learning.
Real-world example
Transformer embeddings becoming overly similar across tokens.
Common mistakes
- Assuming deeper networks always increase representational diversity.
Follow-up questions
- How is rank collapse detected?