What is the difference between dense and sparse attention mechanisms?

Dense attention computes all token interactions; sparse attention restricts interactions to subsets.

Dense attention has O(n²) complexity, while sparse attention reduces computation by limiting token interactions via patterns like sliding windows or learned sparsity. Sparse methods trade expressiveness for efficiency.