GPT-3: Language Models are Few-Shot Learners
An AI-generated review of GPT-3, the 175-billion-parameter language model that showed scale alone can unlock in-context learning without any gradient updates.
4 reports · 2020–2023
Scaling laws, compute-optimal training and the large-model results that reframed what a language model can do.
An AI-generated review of GPT-3, the 175-billion-parameter language model that showed scale alone can unlock in-context learning without any gradient updates.
Kaplan et al. showed language model loss follows predictable power laws in parameters, data and compute — the empirical rules behind modern scaling.
Chinchilla proved a 70B model on 1.4T tokens beats a 280B model on the same compute — most large models were badly undertrained on data.
LLaMA showed a 13B model trained on a trillion tokens could beat GPT-3 175B, reframing the scaling debate around compute-optimal token budgets.
Narrated films that touch this topic, with chapters, captions and transcripts.
1:40Explainer · 4 chapters
Why a language model feels like it understands, what it is actually doing, and three questions that replace “does it understand?”.