4 reports · 2020–2023

Foundations & scaling

Scaling laws, compute-optimal training and the large-model results that reframed what a language model can do.

Fig. 02 / Field notesThe shape of diminishing returns
An illustrative scaling curve: loss falls with increasing compute, with diminishing returns. Axes are schematic.
A schematic loss curve makes a scaling relationship visible.

GPT-3: Language Models are Few-Shot Learners

Foundations & scaling · 9 min read · Advanced

An AI-generated review of GPT-3, the 175-billion-parameter language model that showed scale alone can unlock in-context learning without any gradient updates.

gpt-3llmsfew-shot-learning Read

Scaling Laws for Neural Language Models

Foundations & scaling · 5 min read · Intermediate

Kaplan et al. showed language model loss follows predictable power laws in parameters, data and compute — the empirical rules behind modern scaling.

scaling-lawslarge-language-modelstraining Read

Chinchilla: Compute-Optimal Training

Foundations & scaling · 5 min read · Intermediate

Chinchilla proved a 70B model on 1.4T tokens beats a 280B model on the same compute — most large models were badly undertrained on data.

scaling-lawstrainingchinchilla Read

LLaMA: Open and Efficient Foundation Language Models

Foundations & scaling · 7 min read · Intermediate

LLaMA showed a 13B model trained on a trillion tokens could beat GPT-3 175B, reframing the scaling debate around compute-optimal token budgets.

large-language-modelsscaling-lawstraining Read

Videos on Foundations & scaling

Narrated films that touch this topic, with chapters, captions and transcripts.

All videos