FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
FlashAttention tiles exact attention in fast on-chip memory, cutting memory from quadratic to linear and making long-context transformers practical.
Explainer · 4:58 · 13 chapters
How a language model runs on hardware you own: tokens, memory bandwidth, quantization, the KV cache, runtimes and scaling out, in five minutes.
This five-minute visual guide explains local LLM inference: what happens when a trained language model runs on hardware you own, and which numbers decide whether it feels fast. It starts with tokens, the context window and the two phases of every answer — prefill, which reads the whole prompt in parallel, and decode, which writes one token at a time — and then shows why decode speed follows memory bandwidth, from 100 GB/s for DDR5 to 3.35 TB/s for an H100.
The middle of the film covers what makes a model fit and run: the memory ladder from registers to SSD, GPU kernels and FlashAttention, the roofline that separates memory-bound decode from compute-bound prefill, quantization from FP16 down to 4-bit, a table of which model sizes fit in 8 to 128 GB, dense versus mixture-of-experts models, and the KV cache that grows with context and batch size.
The last third is practical: seven runtimes — llama.cpp, Ollama, LM Studio, vLLM, MLX, TensorRT-LLM and Exo — and how to choose between them by hardware, the two numbers that describe how a model feels (time to first token and tokens per second), and what changes when one machine is not enough: tensor and pipeline parallelism and the interconnect speeds that decide the result. It ends on a six-item checklist: hardware, memory, kernels, quantization, runtime and scale. Where the film shows an illustrative curve or reading rather than a measurement, such as the perplexity curve and the time-to-first-token dial, it labels it that way on screen.
Spoken or shown on screen, gathered here so you can scan them and jump to the moment each one appears.
| Memory | Bandwidth in the film | Watch |
|---|---|---|
| DDR5 system memory | 100 GB/s | |
| Apple M4 Pro | 273 GB/s | |
| NVIDIA DGX Spark | 273 GB/s (128 GB unified memory) | |
| NVIDIA RTX 4090 | 1 TB/s | |
| NVIDIA H100 | 3.35 TB/s |
| Link | Bandwidth in the film | Watch |
|---|---|---|
| PCIe 5.0 x16 | 64 GB/s | |
| Thunderbolt 5 | 10 GB/s | |
| 200 Gb Ethernet | 25 GB/s | |
| NVLink | 900 GB/s |
| What | Size in the film | Watch |
|---|---|---|
| 8B model at FP16 | 16 GB | |
| The same model as Q4_K_M | about 4.8 GB | |
| 7B model at Q4 | 4.4 GB of weights (on screen) | |
| 70B model at Q4 | 40 GB of weights (on screen) | |
| 70B model at FP16 | 140 GB of weights (on screen) | |
| KV cache at 8,000 tokens | about 1 GB per sequence | |
| KV cache for a batch of 8 | 8 GB |
| Memory | Model targets | Note on screen | Watch |
|---|---|---|---|
| 8 GB | 3–4B at Q4 | Narrated example: 7B Q4 | |
| 16 GB | 8B at Q4 or 14B at Q4 | 7B Q4: longer context | |
| 24 GB | 14B at Q4 or 32B at Q4 (tight) | 13B Q4: room for cache | |
| 48–64 GB | 70B at Q4 | 48 GB is tight; 64 GB is safer | |
| 128 GB (unified) | about 120B at Q4; 70B at FP16: no fit | FP16 weights alone: 140 GB |
Editor's note. At 2:52 the narration says a 70B model fits “at full precision” in 128 GB. The on-screen table at that moment says the opposite: 70B at FP16 does not fit, because its weights alone are 140 GB (70 billion parameters × 2 bytes). The table is the figure to trust.
The film advises keeping 20% of memory free; its budget bars mark an 80% safe limit.
Reproduced as stated in the film. Know Your ML has not re-checked these figures against vendor datasheets, so read the source before you quote a number. The perplexity curve and the 0.42-second time-to-first-token dial are labelled illustrative on screen, so they are left out.
Inference means using a trained model rather than training one. Local inference runs that model on hardware you own, so prompts stay private, costs stay fixed and it works offline.
Watch this part
Decode writes one token at a time and reloads every weight for each token, so it is memory-bound and its speed follows memory bandwidth: 100 GB/s for DDR5, 273 GB/s for an M4 Pro, 1 TB/s for an RTX 4090, 3.35 TB/s for an H100. Prefill handles many tokens at once and is compute-bound.
Watch this part
The film's rule is parameters × bits ÷ 8, plus KV cache and overhead, with 20% of memory kept free. An 8B model needs 16 GB at FP16 but about 4.8 GB as Q4_K_M, and a 7B model at Q4 has 4.4 GB of weights, which fits in 8 GB.
Watch this part
Each token keeps its keys and values so the model never recomputes them. The size is 2 × layers × KV heads × head width × context × batch × bytes: about 1 GB per sequence at 8,000 tokens and 8 GB for a batch of 8. Grouped-query attention shrinks it several-fold, and prefix caching reuses a shared prompt's entries.
Watch this part
The film says to choose by hardware, not by hype: Ollama or llama.cpp on a laptop, MLX or Ollama on a Mac, vLLM on one NVIDIA GPU, and vLLM or TensorRT-LLM across several GPUs.
Watch this part
Time to first token is how long you wait for the first word; tokens per second is how fast the rest arrives. Throughput counts everyone's tokens, latency counts yours, and continuous batching lets new requests join a running batch.
Watch this part
When the model does not fit or you want more throughput. Tensor parallelism splits each layer across devices and pipeline parallelism gives each device a block of layers. Scale-out buys capacity and throughput but costs bandwidth and latency: a 200 Gb Ethernet link moves 25 GB/s, which the film calls ten times slower than a node's own memory.
Watch this part
Inference means using a trained model, not training one. Every time a chatbot answers, that is inference happening. Local inference runs that model on hardware you own. Prompts stay private, costs stay fixed, and it works offline.
Text becomes tokens, small chunks of characters or words. The context window is how many tokens fit in memory. The work then happens in two phases. Prefill, the first phase, reads the whole prompt in parallel. Decode, the second phase, writes one token at a time.
A CPU has few fast cores; a GPU has thousands. Apple Silicon uses unified memory for both. An NPU speeds narrow work at low power. Decode speed follows memory bandwidth closely. DDR5 moves 100 gigabytes per second. An M4 Pro moves 273 gigabytes per second. An RTX 4090 moves 1 terabyte per second; an H100 moves 3.35 terabytes.
Memory is a ladder, from fast and tiny to slow and huge. Registers and SRAM sit on the chip. L2 is shared; HBM is the workspace. DRAM is system memory; SSDs store weights. If weights don't fit, layers stream over PCIe. PCIe moves 64 gigabytes per second. Offloading layers costs tokens per second.
The GPU runs thousands of tiny programs called kernels. Matrix multiply dominates, so kernels tile it into blocks. Attention is softmax of Q times K transpose, over root dk. Softmax turns scores into percentages summing to one. RoPE encodes position; LayerNorm stabilizes activations. Kernel fusion merges steps into one memory trip. FlashAttention does that, keeping tiles on chip.
A roofline chart plots speed against arithmetic intensity. Intensity is math performed per byte loaded. Below the ridge, a workload is memory-bound. Above the ridge, it becomes compute-bound. Prefill handles many tokens at once, so it is compute-bound. Decode reloads every weight per token, so it is memory-bound.
Quantization stores each weight in fewer bits. FP16 uses two bytes, INT8 one, INT4 half a byte. An 8-billion-parameter model needs 16 gigabytes at FP16. Q4_K_M cuts that to about 4.8 gigabytes. Perplexity measures the error, and flattens near 4 bits. GGUF file names show the scheme and bit width.
At 8 gigabytes, a 4-bit 7B fits. At 16, the same model, longer context. At 24 gigabytes, a 4-bit 13B. At 48 to 64, a quantized 70B. At 128, 70B at full precision. Dense models use all their parameters. Mixture-of-experts stores 100 billion and activates 10.
Each token keeps keys and values: the KV cache. Size is two, times layers, times KV heads, times head width. Times context, times batch, times bytes. 8,000 tokens: a gigabyte per sequence. A batch of 8: 8 gigabytes. Grouped-query attention shrinks that cache several-fold. Prefix caching reuses a shared prompt's entries.
Runtimes run models on your own machine. llama.cpp is the lean C++ engine behind GGUF. Ollama and LM Studio wrap it in apps. vLLM serves many users with PagedAttention and continuous batching. MLX targets Apple Silicon; TensorRT-LLM targets NVIDIA. Exo spreads one model across several machines. Choose by hardware, not by hype.
Two numbers describe how it feels. Time to first token is how long you wait for the first word. Tokens per second is how fast the rest arrives. Throughput counts everyone's tokens; latency counts yours. Continuous batching lets new requests join a running batch.
Tensor parallelism splits each layer across devices. Pipeline parallelism gives each device layers. The interconnect speed decides the result. PCIe moves 64 gigabytes per second. Thunderbolt 5 gives 10, 200-gigabit Ethernet 25. NVLink reaches 900 gigabytes per second. DGX Spark: 128 gigabytes, 273 per second. Two to four nodes link via ConnectX-7. Mac minis cluster over Thunderbolt 5 with Exo.
Local inference is weights plus cache, on hardware you own. Bandwidth sets the speed; quantization sets the fit. Kernels and runtimes do the heavy lifting. One machine is often enough, and clusters extend it.
The transcript is the film's own subtitle track, with spoken numbers written as numerals. Click any sentence to play from there.
FlashAttention tiles exact attention in fast on-chip memory, cutting memory from quadratic to linear and making long-context transformers practical.
QLoRA fine-tunes 65B models on one 48GB GPU by combining 4-bit quantization with LoRA adapters — nearly matching full-precision tuning.
Mixtral routes each token to 2 of 8 experts, matching LLaMA-2 70B quality with 6x faster inference — open sparse MoE that actually works.
The Transformer replaced recurrence with self-attention, enabling massively parallel training — the architecture behind GPT, BERT and modern LLMs.
Rotary position embeddings encode token order as rotations in embedding space, giving transformers relative positions that stretch to longer text.
LLaMA showed a 13B model trained on a trillion tokens could beat GPT-3 175B, reframing the scaling debate around compute-optimal token budgets.