Scaling Laws for Neural Language Models
Kaplan et al. showed language model loss follows predictable power laws in parameters, data and compute — the empirical rules behind modern scaling.
8 steps · about 56 min · 2020–2024
Scaling laws and compute-optimal training set the budget; adapters, quantization, flash attention and sparse experts decide how it is spent at inference time.
Prefer to jump around? Each step is a standalone report with its own citations, figures and FAQ.
Kaplan et al. showed language model loss follows predictable power laws in parameters, data and compute — the empirical rules behind modern scaling.
Chinchilla proved a 70B model on 1.4T tokens beats a 280B model on the same compute — most large models were badly undertrained on data.
LoRA adapts large models by training small low-rank matrices while freezing the base weights — fine-tuning quality at a fraction of the cost.
QLoRA fine-tunes 65B models on one 48GB GPU by combining 4-bit quantization with LoRA adapters — nearly matching full-precision tuning.
FlashAttention tiles exact attention in fast on-chip memory, cutting memory from quadratic to linear and making long-context transformers practical.
The Switch Transformer scaled to a trillion parameters with simplified mixture-of-experts routing, activating one expert per token for efficient sparsity.
Mixtral routes each token to 2 of 8 experts, matching LLaMA-2 70B quality with 6x faster inference — open sparse MoE that actually works.
A synthetic proposal for energy-aware MoE routing that trades a little accuracy for large power savings in sparse expert models.
Narrated films that cover the same ground, with chapters, captions and transcripts.
4:58Explainer · 13 chapters
How a language model runs on hardware you own: tokens, memory bandwidth, quantization, the KV cache, runtimes and scaling out, in five minutes.
8:53Essay film · 10 chapters
Five common arguments against AI, taken seriously one at a time and tested against history and published data — plus what the critics get right.