How does attention routing reduce compute cost in large-scale transformer inference systems?

Attention routing selectively computes attention only over relevant token subsets instead of full quadratic attention.

Attention routing is an optimization where the model dynamically selects which tokens should interact during self-attention. Instead of computing full pairwise attention, routing mechanisms identify relevant token subsets using learned gates, clustering, or heuristics.

This reduces computational complexity and improves efficiency in long-context scenarios. It is especially useful in systems where most tokens are not equally relevant to each other.

Routing can be static (predefined patterns) or dynamic (learned during inference), and it is often combined with sparse attention methods.