How do transformers work in PyTorch at a high level?

Transformers use self-attention to model relationships between tokens.

They compute query, key, value matrices and apply scaled dot-product attention to capture global dependencies.