What is multi-head attention in Vision Transformers?

Multi-head attention allows a model to attend to different representation subspaces simultaneously.

Multi-head attention splits embeddings into multiple heads, each performing scaled dot-product attention independently. These heads learn different relationships (spatial patterns, textures, global context). Outputs are concatenated and linearly transformed, enabling richer representation learning than single-head attention.