What is cross-attention in multimodal vision-language models?

Cross-attention aligns information between two modalities like text and images.

Cross-attention allows one modality (e.g., text tokens) to attend to another modality (e.g., image patches). Queries come from one modality while keys and values come from another. This mechanism is fundamental in vision-language models such as CLIP-based systems and diffusion models with text conditioning.