How do foundation models use unsupervised pretraining at scale?

They learn general-purpose representations from massive unlabeled datasets using self-supervised objectives.

Foundation models like large language models and vision transformers are trained on massive corpora using unsupervised or self-supervised objectives such as next-token prediction, masked modeling, or contrastive objectives. The architecture learns hierarchical representations that transfer across tasks. Scaling laws show performance improves predictably with model size, data, and compute.