What is self-attention complexity problem in Vision Transformers and how is it solved?

Self-attention has quadratic complexity in number of tokens, making it expensive for high-resolution images.

In Vision Transformers, self-attention computes pairwise interactions between all patches, resulting in O(N²) complexity. For high-resolution images, N becomes large, causing memory and compute bottlenecks. Solutions include windowed attention (Swin Transformer), sparse attention, linear attention approximations, and patch merging strategies.