How do large language models scale computationally?

LLMs scale by increasing parameters, data, and compute following scaling laws.

Scaling laws show that performance improves predictably with more parameters, data, and compute. However, training cost grows quadratically in attention layers, requiring distributed training and optimization techniques like model parallelism.