How does attention mechanism work internally in ChatGPT?

Attention allows the model to weigh relevance of all tokens when generating each output token.

Self-attention computes relationships between all tokens in a sequence. Each token is transformed into query, key, and value vectors. The dot product between queries and keys determines relevance scores, which are normalized using softmax. These scores weight the value vectors to produce context-aware representations.

This mechanism allows ChatGPT to capture long-range dependencies in text, unlike RNNs. Multi-head attention further improves expressiveness by learning different representation subspaces.