What is the mathematical intuition behind self-attention as a kernel function?

Self-attention can be interpreted as a learned similarity kernel over token embeddings.

Self-attention computes pairwise similarity between queries and keys, effectively acting as a dynamic kernel function. Unlike fixed kernels in classical ML, transformer kernels are learned and context-dependent. This allows adaptive similarity metrics that change based on input sequence semantics.