How do transformer-based models learn unsupervised representations?

Transformers learn unsupervised representations using masked prediction or next-token prediction.

Models like BERT use masked language modeling, while GPT uses autoregressive prediction. These objectives force transformers to learn contextual embeddings without explicit labels. Attention mechanisms capture long-range dependencies, making them powerful for representation learning.