GPT-3: Language Models are Few-Shot Learners
An AI-generated review of GPT-3, the 175-billion-parameter language model that showed scale alone can unlock in-context learning without any gradient updates.
Primary source: Language Models are Few-Shot Learners (arXiv:2005.14165)
GPT-3, released in 2020 by OpenAI, is the paper that made "scale is all you need" a mainstream research program. A 175-billion-parameter transformer trained on roughly 45TB of text, it demonstrated that a language model could perform a wide range of tasks it had never been trained on — translation, question answering, arithmetic, code generation — using only a handful of examples supplied in its prompt, with no weight updates at all. The paper's most consequential claim: in-context learning emerges from scale.
The core idea
The authors systematically compared three ways to adapt a language model to a task:
- Fine-tuning — gradient updates on task data (best accuracy, but needs labels and compute).
- Few-shot — the model is given a natural-language instruction plus a few input-output examples in the prompt; no training happens.
- Zero-shot / one-shot — the instruction only, or instruction plus a single example.
The paper's headline is the few-shot setting: purely by conditioning on examples in the context, GPT-3 matches or approaches fine-tuned models of similar size on many tasks — and in a handful of cases (trivia QA, some translation pairs) actually exceeds them. It also documents how performance improves smoothly with model size and how few-shot capability appears primarily at the largest scales, suggesting an emergent, scale-driven phenomenon.
How it works
- Architecture: a standard decoder-only transformer with alternating dense and sparse (locally banded) attention layers — a design shared with GPT-2's and GPT-3's sparse attention variant. 96 layers, 96 attention heads, 12,288 hidden units.
- Data: Common Crawl (filtered and deduplicated), WebText2, books, and Wikipedia — ~300B tokens, with high-quality corpora upsampled. Explicit deduplication against downstream test sets prevented benchmark contamination.
- Training: ~10³ FLOP/s-days, roughly 30x GPT-2's compute. The paper's scaling curves predicted the final model's loss well, validating the scaling-law methodology from the earlier GPT-3-era work.
- Sampling: nucleus sampling with temperature schedules for generation tasks; greedy decoding for others.
Why it matters
- In-context learning as a capability. GPT-3 showed that a model can learn "at inference time" from examples in the prompt — the foundation of every modern prompt-engineering workflow and of few-shot agent behavior.
- Scaling law evidence. It made the case that model size, data, and compute follow predictable power laws, directly inspiring the compute-intensive scaling race of the following years.
- Broad capability without task-specific training. Arithmetic, translation, QA, and even novel task formats (like word unscrambling) worked few-shot — a shock at the time.
- Economic implications. It demonstrated that a single general model could substitute for many specialized pipelines, reshaping how AI products were built.
Limitations and open questions
The paper is unusually candid about failure modes: GPT-3 struggles with common-sense physical reasoning, factual consistency (it generates plausible but false statements), and biases — it reproduces and amplifies stereotypes present in its training data. Its 175B parameters made inference expensive and research access exclusive, concentrating follow-up work in well-resourced labs. The paper also raised unresolved questions about benchmark contamination, the true nature of in-context learning, and whether "few-shot learning" is learning at all rather than sophisticated pattern recall.
Key takeaways
- GPT-3 = 175B-parameter decoder-only transformer trained on ~300B tokens of web text.
- Few-shot prompting — instruction + a few examples in context, no gradient updates — unlocks broad task performance.
- Scale improves smooth, predictable ways; few-shot capability is most visible at the largest sizes.
- The paper's candid failure analysis (reasoning gaps, hallucination, bias, cost) framed the safety and alignment agenda that followed.
Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.
Cite this report
@misc{knowyourml-gpt-3-few-shot-learners
title = {GPT-3: Language Models are Few-Shot Learners},
author = {{Know Your ML}},
year = {2020},
howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=gpt-3-few-shot-learners}},
note = {AI-generated review of arXiv:2005.14165. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit
Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.
References
- Language Models are Few-Shot Learners (arXiv:2005.14165) — https://arxiv.org/abs/2005.14165
Part of
Beyond the paper
How to Remain Valuable When Intelligence Becomes Cheap
This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?