Knowledge Distillation: Teacher-Student Compression
Knowledge distillation compresses big teacher models into small fast students using soft targets — the key to efficient edge AI.
7 reports · 2015–2024
Doing more with less compute: adapters, quantisation, fast attention, sparse experts and deployment trade-offs.
Knowledge distillation compresses big teacher models into small fast students using soft targets — the key to efficient edge AI.
LoRA adapts large models by training small low-rank matrices while freezing the base weights — fine-tuning quality at a fraction of the cost.
The Switch Transformer scaled to a trillion parameters with simplified mixture-of-experts routing, activating one expert per token for efficient sparsity.
FlashAttention tiles exact attention in fast on-chip memory, cutting memory from quadratic to linear and making long-context transformers practical.
A synthetic proposal for energy-aware MoE routing that trades a little accuracy for large power savings in sparse expert models.
QLoRA fine-tunes 65B models on one 48GB GPU by combining 4-bit quantization with LoRA adapters — nearly matching full-precision tuning.
Mixtral routes each token to 2 of 8 experts, matching LLaMA-2 70B quality with 6x faster inference — open sparse MoE that actually works.
Narrated films that touch this topic, with chapters, captions and transcripts.
4:58Explainer · 13 chapters
How a language model runs on hardware you own: tokens, memory bandwidth, quantization, the KV cache, runtimes and scaling out, in five minutes.
8:53Essay film · 10 chapters
Five common arguments against AI, taken seriously one at a time and tested against history and published data — plus what the critics get right.