What are compute-optimal scaling laws in NLP?

They define optimal balance between model size, dataset size, and compute budget.

Scaling laws show that performance improves predictably with compute, but only if data and parameters are balanced. Training too large models on insufficient data leads to underperformance. Compute-optimal training finds the best ratio between dataset size and model parameters.