AI-generated review 5 min read Intermediate 851 words

Scaling Laws for Neural Language Models

Kaplan et al. showed language model loss follows predictable power laws in parameters, data and compute — the empirical rules behind modern scaling.

scaling-laws large-language-models training compute foundations
Foundations & scaling · 2020

Primary source: Scaling Laws for Neural Language Models (arXiv:2001.08361)

In 2020, Jared Kaplan and colleagues at OpenAI trained dozens of transformer language models at different sizes and plotted the results. The points fell on smooth curves: test loss decreased as a power law in model size, dataset size, and compute. Bigger was better, and — more importantly — predictably better.

The three power laws

The paper reports three scaling relationships, each holding over several orders of magnitude:

  • Parameters (N): loss scales as roughly N^-0.07 when data is not a bottleneck.
  • Data (D): loss scales as roughly D^-0.09 with a large enough model.
  • Compute (C): loss scales as roughly C^-0.05 when training efficiently.

In plain terms: to cut loss by a fixed amount, you must multiply your budget by a fixed factor. There are no sudden jumps and no plateaus within the measured range — just a straight line on a log-log plot.

Loss vs compute (log-log) Training compute (FLOPs) Test loss 1M params 1.5B params
On a log-log plot, loss falls on a straight line as compute grows — the signature of a power law.

Bigger models are also more sample-efficient: they reach any given loss with fewer training steps and fewer tokens seen. Scale does not just raise the ceiling; it shortens the climb.

How the fits were done

The methodology matters because it is what made the claims trustworthy. The authors trained models from 768K to 1.5B parameters on datasets up to 23B tokens, smoothing loss curves to remove optimizer noise and — importantly — excluding embedding parameters from N, since embeddings scale differently from the rest of the network. Each law was fitted while holding the other two inputs effectively infinite, isolating one bottleneck at a time.

They also separated reducible loss from the irreducible entropy floor of the text itself. Power laws describe the reducible part: no amount of scale pushes loss below the entropy of language, and the fits estimate that floor at roughly 1.7 nats per token on their distribution.

Reading the exponents

The exponents encode a budgeting priority. Because loss falls fastest in data (D^-0.09) and slowest in raw compute when misallocated, the optimal strategy grows everything together: the paper's headline recommendation was that an 8x compute increase should buy roughly 5x the parameters and 2x the data, with the rest going to longer training.

This is also where the 2020 analysis went wrong in retrospect. Its small models were undertrained, which made data look less valuable than it is. Chinchilla reran the methodology with properly trained models at every scale and found the data exponent effectively larger — flipping the recommendation to equal scaling. The power-law form held; the fitted constants did not.

What matters less than scale

One of the paper's striking claims is that architectural details — the number of layers versus width, the attention head count — have only second-order effects. Within a broad family of transformer shapes, performance is mostly a function of total parameters, not how they are arranged.

Overfitting follows a predictable rule too. Small models stop improving once they have seen enough data; the paper estimates that model size and dataset size should grow together, and that training past the point where loss stops falling just wastes compute.

What the laws do not predict

Kaplan's laws describe test loss, not downstream task accuracy. Later work showed that capabilities like multi-step reasoning can appear abruptly even while loss falls smoothly, so the laws predict the training curve better than they predict what the model can do. A 2022 study of "emergent abilities" catalogued over a hundred tasks where performance jumps discontinuously with scale — the loss curve gives no warning.

They also say nothing about data quality. The laws were fitted on a fixed web-text distribution; curated, deduplicated, or synthetic data shifts the constants, sometimes dramatically. Practitioners treat the exponents as a starting prior and refit on their own mixture — which is why every serious lab now runs its own small-scale scaling studies before committing to a flagship training run.

The paper also recommended scaling parameters faster than data — a conclusion the Chinchilla study overturned two years later by showing most large models were badly undertrained. The power-law form survived; the recommended ratio did not.

Why it matters

Before scaling laws, training a 10x larger model was a gamble. After them, it was a budget line item: labs could predict the loss of a model they had not trained yet and decide whether the compute bill was worth it. GPT-3, PaLM, and every frontier training run since were sized using this logic.

Bottom line

Language model loss falls as a smooth power law in parameters, data, and compute. The 2020 paper made scale predictable, turned training runs into engineering plans, and set the terms of every scaling debate that followed.

Sources

  • Kaplan et al., Scaling Laws for Neural Language Models, 2020. arXiv:2001.08361
  • Hoffmann et al., Training Compute-Optimal Large Language Models, 2022. arXiv:2203.15556

Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.

Cite this report
@misc{knowyourml-scaling-laws-neural-language-models
  title        = {Scaling Laws for Neural Language Models},
  author       = {{Know Your ML}},
  year         = {2020},
  howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=scaling-laws-neural-language-models}},
  note         = {AI-generated review of arXiv:2001.08361. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit — decision matrix, scorecards and trade-off analysis
Toolkit 7 decision tools Model-agnostic

Machine Learning Decision Toolkit

Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.

$24 View on Gumroad

References

Frequently asked questions

What exactly is the scaling law relationship?

Loss falls as a power law in each input: roughly L(N) ~ N^-0.07 for parameters, L(D) ~ D^-0.09 for data, and L(C) ~ C^-0.05 for compute. Double the compute and loss drops by a predictable few percent.

Does architecture matter, or only scale?

Within the tested range, architecture details changed loss far less than scale did. A bigger model reliably beat a smaller one regardless of modest design differences.

How to Remain Valuable When Intelligence Becomes Cheap — book cover
eBook 224 pages

How to Remain Valuable When Intelligence Becomes Cheap

This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?

Read the eBook