What are theoretical limitations of attention mechanisms?

Attention is limited by quadratic complexity, limited inductive bias, and weak positional reasoning.

Self-attention provides flexible token interactions but lacks strong inductive biases like locality or hierarchy. Its O(n²) complexity limits scalability. Additionally, attention is permutation-invariant without positional encoding, and struggles with algorithmic reasoning without training signal. These limitations motivate sparse attention, recurrence, and hybrid architectures.