How does attention scaling behave mathematically with sequence length?

Attention complexity scales as O(n²) in both compute and memory.

Self-attention computes pairwise token interactions, requiring matrix multiplication between Q and Kᵀ. This leads to quadratic scaling in sequence length. Approximations like linear attention reduce complexity but trade off expressiveness.