Explainer · 4:58 · 13 chapters

Local LLM Inference, Explained

How a language model runs on hardware you own: tokens, memory bandwidth, quantization, the KV cache, runtimes and scaling out, in five minutes.

0:00 What local inference is 4:58

Download MP4 (21.9 MB) Captions (.vtt)

About this video

This five-minute visual guide explains local LLM inference: what happens when a trained language model runs on hardware you own, and which numbers decide whether it feels fast. It starts with tokens, the context window and the two phases of every answer — prefill, which reads the whole prompt in parallel, and decode, which writes one token at a time — and then shows why decode speed follows memory bandwidth, from 100 GB/s for DDR5 to 3.35 TB/s for an H100.

The middle of the film covers what makes a model fit and run: the memory ladder from registers to SSD, GPU kernels and FlashAttention, the roofline that separates memory-bound decode from compute-bound prefill, quantization from FP16 down to 4-bit, a table of which model sizes fit in 8 to 128 GB, dense versus mixture-of-experts models, and the KV cache that grows with context and batch size.

The last third is practical: seven runtimes — llama.cpp, Ollama, LM Studio, vLLM, MLX, TensorRT-LLM and Exo — and how to choose between them by hardware, the two numbers that describe how a model feels (time to first token and tokens per second), and what changes when one machine is not enough: tensor and pipeline parallelism and the interconnect speeds that decide the result. It ends on a six-item checklist: hardware, memory, kernels, quantization, runtime and scale. Where the film shows an illustrative curve or reading rather than a measurement, such as the perplexity curve and the time-to-first-token dial, it labels it that way on screen.

Key takeaways

  • Hardware. Start with the machine you already own. One computer is often enough, and a cluster only extends it when the model does not fit or you need more throughput.
  • Memory. Bandwidth sets the speed. Decode reloads every weight for every token, so tokens per second tracks memory bandwidth — 100 GB/s for DDR5, 273 GB/s for an M4 Pro, 1 TB/s for an RTX 4090.
  • Kernels. The heavy lifting. Matrix multiplication dominates GPU work, and fused kernels such as FlashAttention keep tiles on chip to avoid extra trips to memory.
  • Quantization. Sets the fit. Fewer bits per weight shrink an 8B model from 16 GB at FP16 to about 4.8 GB as Q4_K_M. Plan memory as parameters × bits ÷ 8, plus KV cache and overhead, and keep 20% free.
  • Runtime. Schedules the kernels. Choose by hardware, not by hype: Ollama or llama.cpp on a laptop, MLX or Ollama on a Mac, vLLM on NVIDIA GPUs.
  • Scale. Only when you must. Scaling out buys capacity and throughput but costs bandwidth and latency; a 200 Gb link moves 25 GB/s, which the film calls ten times slower than a node's own memory.

Chapters

Numbers from the film

Spoken or shown on screen, gathered here so you can scan them and jump to the moment each one appears.

Memory bandwidth: decode speed follows it
Memory Bandwidth in the film Watch
DDR5 system memory 100 GB/s
Apple M4 Pro 273 GB/s
NVIDIA DGX Spark 273 GB/s (128 GB unified memory)
NVIDIA RTX 4090 1 TB/s
NVIDIA H100 3.35 TB/s
Links between memory pools and machines
Link Bandwidth in the film Watch
PCIe 5.0 x16 64 GB/s
Thunderbolt 5 10 GB/s
200 Gb Ethernet 25 GB/s
NVLink 900 GB/s
Weights and cache sizes
What Size in the film Watch
8B model at FP16 16 GB
The same model as Q4_K_M about 4.8 GB
7B model at Q4 4.4 GB of weights (on screen)
70B model at Q4 40 GB of weights (on screen)
70B model at FP16 140 GB of weights (on screen)
KV cache at 8,000 tokens about 1 GB per sequence
KV cache for a batch of 8 8 GB
What fits in how much memory (the film's on-screen table)
Memory Model targets Note on screen Watch
8 GB 3–4B at Q4 Narrated example: 7B Q4
16 GB 8B at Q4 or 14B at Q4 7B Q4: longer context
24 GB 14B at Q4 or 32B at Q4 (tight) 13B Q4: room for cache
48–64 GB 70B at Q4 48 GB is tight; 64 GB is safer
128 GB (unified) about 120B at Q4; 70B at FP16: no fit FP16 weights alone: 140 GB

Editor's note. At 2:52 the narration says a 70B model fits “at full precision” in 128 GB. The on-screen table at that moment says the opposite: 70B at FP16 does not fit, because its weights alone are 140 GB (70 billion parameters × 2 bytes). The table is the figure to trust.

The film advises keeping 20% of memory free; its budget bars mark an 80% safe limit.

Reproduced as stated in the film. Know Your ML has not re-checked these figures against vendor datasheets, so read the source before you quote a number. The perplexity curve and the 0.42-second time-to-first-token dial are labelled illustrative on screen, so they are left out.

Quick answers

What is local LLM inference?

Inference means using a trained model rather than training one. Local inference runs that model on hardware you own, so prompts stay private, costs stay fixed and it works offline.

Watch this part

Why is generating text limited by memory bandwidth?

Decode writes one token at a time and reloads every weight for each token, so it is memory-bound and its speed follows memory bandwidth: 100 GB/s for DDR5, 273 GB/s for an M4 Pro, 1 TB/s for an RTX 4090, 3.35 TB/s for an H100. Prefill handles many tokens at once and is compute-bound.

Watch this part

How much memory does a local model need?

The film's rule is parameters × bits ÷ 8, plus KV cache and overhead, with 20% of memory kept free. An 8B model needs 16 GB at FP16 but about 4.8 GB as Q4_K_M, and a 7B model at Q4 has 4.4 GB of weights, which fits in 8 GB.

Watch this part

What is the KV cache and how big does it get?

Each token keeps its keys and values so the model never recomputes them. The size is 2 × layers × KV heads × head width × context × batch × bytes: about 1 GB per sequence at 8,000 tokens and 8 GB for a batch of 8. Grouped-query attention shrinks it several-fold, and prefix caching reuses a shared prompt's entries.

Watch this part

Which runtime should I use for local models?

The film says to choose by hardware, not by hype: Ollama or llama.cpp on a laptop, MLX or Ollama on a Mac, vLLM on one NVIDIA GPU, and vLLM or TensorRT-LLM across several GPUs.

Watch this part

What are time to first token and tokens per second?

Time to first token is how long you wait for the first word; tokens per second is how fast the rest arrives. Throughput counts everyone's tokens, latency counts yours, and continuous batching lets new requests join a running batch.

Watch this part

When is one machine not enough?

When the model does not fit or you want more throughput. Tensor parallelism splits each layer across devices and pipeline parallelism gives each device a block of layers. Scale-out buys capacity and throughput but costs bandwidth and latency: a 200 Gb Ethernet link moves 25 GB/s, which the film calls ten times slower than a node's own memory.

Watch this part

Transcript

Read the full transcript612 words

What local inference is

Inference means using a trained model, not training one. Every time a chatbot answers, that is inference happening. Local inference runs that model on hardware you own. Prompts stay private, costs stay fixed, and it works offline.

Tokens, context and the two phases

Text becomes tokens, small chunks of characters or words. The context window is how many tokens fit in memory. The work then happens in two phases. Prefill, the first phase, reads the whole prompt in parallel. Decode, the second phase, writes one token at a time.

Processors and memory bandwidth

A CPU has few fast cores; a GPU has thousands. Apple Silicon uses unified memory for both. An NPU speeds narrow work at low power. Decode speed follows memory bandwidth closely. DDR5 moves 100 gigabytes per second. An M4 Pro moves 273 gigabytes per second. An RTX 4090 moves 1 terabyte per second; an H100 moves 3.35 terabytes.

The memory ladder

Memory is a ladder, from fast and tiny to slow and huge. Registers and SRAM sit on the chip. L2 is shared; HBM is the workspace. DRAM is system memory; SSDs store weights. If weights don't fit, layers stream over PCIe. PCIe moves 64 gigabytes per second. Offloading layers costs tokens per second.

Kernels, attention and FlashAttention

The GPU runs thousands of tiny programs called kernels. Matrix multiply dominates, so kernels tile it into blocks. Attention is softmax of Q times K transpose, over root dk. Softmax turns scores into percentages summing to one. RoPE encodes position; LayerNorm stabilizes activations. Kernel fusion merges steps into one memory trip. FlashAttention does that, keeping tiles on chip.

The roofline: memory-bound vs compute-bound

A roofline chart plots speed against arithmetic intensity. Intensity is math performed per byte loaded. Below the ridge, a workload is memory-bound. Above the ridge, it becomes compute-bound. Prefill handles many tokens at once, so it is compute-bound. Decode reloads every weight per token, so it is memory-bound.

Quantization: fewer bits per weight

Quantization stores each weight in fewer bits. FP16 uses two bytes, INT8 one, INT4 half a byte. An 8-billion-parameter model needs 16 gigabytes at FP16. Q4_K_M cuts that to about 4.8 gigabytes. Perplexity measures the error, and flattens near 4 bits. GGUF file names show the scheme and bit width.

Fitting the model in memory

At 8 gigabytes, a 4-bit 7B fits. At 16, the same model, longer context. At 24 gigabytes, a 4-bit 13B. At 48 to 64, a quantized 70B. At 128, 70B at full precision. Dense models use all their parameters. Mixture-of-experts stores 100 billion and activates 10.

The KV cache

Each token keeps keys and values: the KV cache. Size is two, times layers, times KV heads, times head width. Times context, times batch, times bytes. 8,000 tokens: a gigabyte per sequence. A batch of 8: 8 gigabytes. Grouped-query attention shrinks that cache several-fold. Prefix caching reuses a shared prompt's entries.

Seven runtimes, one job

Runtimes run models on your own machine. llama.cpp is the lean C++ engine behind GGUF. Ollama and LM Studio wrap it in apps. vLLM serves many users with PagedAttention and continuous batching. MLX targets Apple Silicon; TensorRT-LLM targets NVIDIA. Exo spreads one model across several machines. Choose by hardware, not by hype.

First token, tokens per second, batching

Two numbers describe how it feels. Time to first token is how long you wait for the first word. Tokens per second is how fast the rest arrives. Throughput counts everyone's tokens; latency counts yours. Continuous batching lets new requests join a running batch.

When one machine is not enough

Tensor parallelism splits each layer across devices. Pipeline parallelism gives each device layers. The interconnect speed decides the result. PCIe moves 64 gigabytes per second. Thunderbolt 5 gives 10, 200-gigabit Ethernet 25. NVLink reaches 900 gigabytes per second. DGX Spark: 128 gigabytes, 273 per second. Two to four nodes link via ConnectX-7. Mac minis cluster over Thunderbolt 5 with Exo.

The whole film in one list

Local inference is weights plus cache, on hardware you own. Bandwidth sets the speed; quantization sets the fit. Kernels and runtimes do the heavy lifting. One machine is often enough, and clusters extend it.

The transcript is the film's own subtitle track, with spoken numbers written as numerals. Click any sentence to play from there.

QLoRA: Quantized Fine-Tuning

Efficiency & deployment · 5 min read · Intermediate

QLoRA fine-tunes 65B models on one 48GB GPU by combining 4-bit quantization with LoRA adapters — nearly matching full-precision tuning.

qloraquantizationfine-tuning Read

Mixtral: Sparse Mixture of Experts

Efficiency & deployment · 5 min read · Intermediate

Mixtral routes each token to 2 of 8 experts, matching LLaMA-2 70B quality with 6x faster inference — open sparse MoE that actually works.

mixture-of-expertsmoeefficiency Read

The Transformer: Attention Without Recurrence

Architectures · 5 min read · Beginner

The Transformer replaced recurrence with self-attention, enabling massively parallel training — the architecture behind GPT, BERT and modern LLMs.

transformersattentionself-attention Read

RoPE: Rotary Position Embeddings

Architectures · 5 min read · Intermediate

Rotary position embeddings encode token order as rotations in embedding space, giving transformers relative positions that stretch to longer text.

position-embeddingsropetransformers Read

LLaMA: Open and Efficient Foundation Language Models

Foundations & scaling · 7 min read · Intermediate

LLaMA showed a 13B model trained on a trillion tokens could beat GPT-3 175B, reframing the scaling debate around compute-optimal token budgets.

large-language-modelsscaling-lawstraining Read