8 steps · about 56 min · 2020–2024

Making models cheap to run

Scaling laws and compute-optimal training set the budget; adapters, quantization, flash attention and sparse experts decide how it is spent at inference time.

Fig. 02 / Reading route8 papers, in order
Reading path: Scaling Laws for Neural Language Models; Chinchilla: Compute-Optimal Training; LoRA: Efficient Fine-Tuning of Large Models; QLoRA: Quantized Fine-Tuning; FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness; Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity; Mixtral: Sparse Mixture of Experts; Energy-Aware Routing in Sparse Mixture-of-Experts Models0102030405060708
Start with Scaling Laws for Neural Language Models. Follow the numbered steps to Energy-Aware Routing in Sparse Mixture-of-Experts Models.
  1. Scaling Laws for Neural Language Models 2020 · 5 min read · Intermediate — Kaplan et al. showed language model loss follows predictable power laws in parameters, data and compute — the empirical rules behind modern scaling.
  2. Chinchilla: Compute-Optimal Training 2022 · 5 min read · Intermediate — Chinchilla proved a 70B model on 1.4T tokens beats a 280B model on the same compute — most large models were badly undertrained on data.
  3. LoRA: Efficient Fine-Tuning of Large Models 2021 · 4 min read · Advanced — LoRA adapts large models by training small low-rank matrices while freezing the base weights — fine-tuning quality at a fraction of the cost.
  4. QLoRA: Quantized Fine-Tuning 2023 · 5 min read · Intermediate — QLoRA fine-tunes 65B models on one 48GB GPU by combining 4-bit quantization with LoRA adapters — nearly matching full-precision tuning.
  5. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness 2022 · 8 min read · Advanced — FlashAttention tiles exact attention in fast on-chip memory, cutting memory from quadratic to linear and making long-context transformers practical.
  6. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity 2021 · 8 min read · Advanced — The Switch Transformer scaled to a trillion parameters with simplified mixture-of-experts routing, activating one expert per token for efficient sparsity.
  7. Mixtral: Sparse Mixture of Experts 2024 · 5 min read · Intermediate — Mixtral routes each token to 2 of 8 experts, matching LLaMA-2 70B quality with 6x faster inference — open sparse MoE that actually works.
  8. Energy-Aware Routing in Sparse Mixture-of-Experts Models 2023 · 16 min read · Advanced — A synthetic proposal for energy-aware MoE routing that trades a little accuracy for large power savings in sparse expert models.

Every step as a full report

Prefer to jump around? Each step is a standalone report with its own citations, figures and FAQ.

Scaling Laws for Neural Language Models

Foundations & scaling · 5 min read · Intermediate

Kaplan et al. showed language model loss follows predictable power laws in parameters, data and compute — the empirical rules behind modern scaling.

scaling-lawslarge-language-modelstraining Read

Chinchilla: Compute-Optimal Training

Foundations & scaling · 5 min read · Intermediate

Chinchilla proved a 70B model on 1.4T tokens beats a 280B model on the same compute — most large models were badly undertrained on data.

scaling-lawstrainingchinchilla Read

LoRA: Efficient Fine-Tuning of Large Models

Efficiency & deployment · 4 min read · Advanced

LoRA adapts large models by training small low-rank matrices while freezing the base weights — fine-tuning quality at a fraction of the cost.

lorafine-tuningparameter-efficiency Read

QLoRA: Quantized Fine-Tuning

Efficiency & deployment · 5 min read · Intermediate

QLoRA fine-tunes 65B models on one 48GB GPU by combining 4-bit quantization with LoRA adapters — nearly matching full-precision tuning.

qloraquantizationfine-tuning Read

Mixtral: Sparse Mixture of Experts

Efficiency & deployment · 5 min read · Intermediate

Mixtral routes each token to 2 of 8 experts, matching LLaMA-2 70B quality with 6x faster inference — open sparse MoE that actually works.

mixture-of-expertsmoeefficiency Read

Watch alongside this path

Narrated films that cover the same ground, with chapters, captions and transcripts.

All videos