How do scaling laws relate model size to cost function minimization?

Scaling laws describe how loss decreases predictably with increased model size, data, and compute.

Empirical scaling laws show that loss decreases as a power law with respect to model parameters, dataset size, and compute. This implies predictable improvements in cost function minimization when scaling systems. However, diminishing returns appear beyond optimal compute/data ratios.