What is attention bottleneck in vision transformers?

Attention bottleneck refers to limitations in information flow caused by restricted token interaction or compression.

In Vision Transformers, attention bottlenecks occur when tokens must compress global information through limited attention heads or reduced token sets. This can restrict expressiveness, especially in long-range dependency modeling. Techniques like multi-scale attention, global tokens, or memory tokens are used to alleviate this bottleneck.