Why do attention mechanisms rely heavily on linear algebra operations?

Attention is matrix multiplication between queries, keys, and values.

Attention computes similarity via dot products (QKᵀ), normalizes using softmax, and applies weighted sums of values. This is purely linear algebra in high-dimensional embedding spaces.