How do transformers perform classification?

Transformers use self-attention and contextual embeddings for classification tasks.

Transformers encode input sequences into contextual representations using multi-head attention. A classification head is added on top of [CLS] token representation.