How does attention interpret long-range dependencies in text?

Attention directly connects distant tokens by computing pairwise relevance scores.

Unlike RNNs, transformers compute direct interactions between all token pairs via dot-product attention. This allows modeling dependencies regardless of distance, but introduces quadratic complexity. Specialized heads often learn to track coreference, syntax, or semantic roles.