How does the curse of dimensionality affect transformer embeddings?

It causes embedding spaces to become overly sparse and distance-uninformative.

Transformer models generate high-dimensional embeddings (often 768–4096 dimensions). In such spaces, vector norms and cosine similarities become tightly concentrated, reducing contrast between semantically different tokens. This leads to challenges in retrieval, clustering, and nearest-neighbor search. Additionally, anisotropy emerges, where embeddings occupy a narrow cone instead of filling space uniformly.