AI-generated review 5 min read Advanced 840 words

Mamba: Selective State Spaces

Mamba replaces attention with selective state spaces that scale linearly in sequence length — matching transformers at several times the inference speed.

mamba state-space-models long-context architectures efficiency
Architectures · 2023

Primary source: Mamba: Linear-Time Sequence Modeling with Selective State Spaces (arXiv:2312.00752)

Attention's weakness is its memory bill: cost grows quadratically with sequence length, and inference must cache every past key and value. The December 2023 Mamba paper asked whether recurrence — the thing transformers killed — could come back in a form fast enough to compete. The answer was selective state-space models: linear-time sequence modeling that matches transformers on language benchmarks while running about 5x faster at inference.

State spaces, briefly

A classical state-space model keeps a hidden state h, updates it with each input (h' = Ah + Bx), and reads out predictions (y = Ch). The whole history compresses into a fixed-size vector, so per-token cost is constant. Earlier structured variants (S4) made this trainable in parallel and handled very long sequences well, but they were time-invariant: the same A, B, C for every token, unable to focus on what matters and ignore the rest.

The selective fix

Mamba's key change is making B, C, and the step size input-dependent. Each token decides how much of the state to keep, overwrite, or ignore — a data-controlled gate that plays the role attention's content-based lookup plays, without pairwise comparisons.

Selectivity breaks the math that made S4 fast (convolution mode), so the paper pairs it with a hardware-aware algorithm: the recurrence runs in fast SRAM with kernel fusion and recomputation, never materializing the full state in slow memory. The result trains about as fast as FlashAttention and infers in linear time with constant memory.

Attention: O(n²) every pair compared Mamba: O(n) fixed state, passed along
Attention compares all pairs; Mamba carries a fixed-size state forward one step at a time.

Selectivity in detail

Three quantities become input-dependent: B (how much the input writes to the state), C (how much the state writes to the output), and Δ (the discretization step — effectively, how much to weigh the present token versus accumulated history). A large Δ resets toward the current input; a small Δ preserves memory. The model learns per-token gating end to end, which is why Mamba can do content-aware reasoning (find the relevant name, ignore filler) that time-invariant S4 could not.

The discretization uses zero-order hold: continuous parameters (A, B) become discrete (Ā, B̄) via Δ, keeping the recurrence stable. A itself stays structured (diagonal, HiPPO-initialized) so the state retains its long-memory bias while B, C, and Δ supply the selectivity.

The systems contribution is easy to underrate. A naive selective scan materializes per-token states in GPU global memory and runs 20–30x slower than optimized attention. Mamba's fused kernel keeps the recurrence in SRAM, recomputes intermediates on the backward pass instead of storing them, and parallelizes across the sequence with an associative scan — matching FlashAttention's training speed while giving linear-time inference.

Mamba-2 and the hybrid turn

Mamba-2 (2024) restricted the state transition to a scalar-times-identity form, which lets the whole layer run as parallel block operations on tensor cores — 2–8x faster training than Mamba-1 with slightly better quality. More consequentially, it framed SSMs and attention as dual views of the same computation (structured masked matrix multiplication), opening the door to hybrids designed from theory rather than experiment.

The hybrid evidence is strong. Jamba interleaves Mamba and attention layers with MoE and fits 256k context in one 80GB GPU. NVIDIA's Nemotron-H and SmolLM3 hybrids beat pure transformers and pure SSMs at matched size. The emerging consensus: SSM layers handle the bulk of the sequence cheaply, while a few attention layers provide exact retrieval. Pure attention is overkill everywhere; pure SSM underperforms somewhere.

What the results showed

Mamba matched or beat same-size transformers on language modeling, commonsense reasoning, and long-context retrieval up to a million tokens in synthetic tests — sequences where attention is computationally hopeless. Throughput was roughly 5x a comparable transformer at generation time, with no KV cache growth. On the Long Range Arena benchmark suite, selective SSMs swept the board, including the Path-X task (16k-length reasoning) where transformers of the same size failed outright.

Adoption followed quickly: Mamba-2 refined the algorithm with a parallel-friendly form, and hybrid models (Mamba layers interleaved with a few attention layers, as in Jamba and NVIDIA's Nemotron-H) took the best of both — linear scaling for the bulk of the sequence, attention for precise retrieval.

Open questions

Pure Mamba still trails attention on tasks that need exact recall of arbitrary earlier tokens ("needle in a haystack" at extreme lengths), because compression is lossy by design. Whether future long-context models are pure SSMs, hybrids, or something else is unresolved — but attention's monopoly is over.

Bottom line

Mamba proved recurrence could be both selective and fast, delivering transformer-quality modeling in linear time. Even where hybrids win, the paper permanently changed the architecture conversation: quadratic attention is now a choice, not a law.

Sources

  • Gu and Dao, Mamba: Linear-Time Sequence Modeling with Selective State Spaces, 2023. arXiv:2312.00752

Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.

Cite this report
@misc{knowyourml-mamba-selective-state-spaces
  title        = {Mamba: Selective State Spaces},
  author       = {{Know Your ML}},
  year         = {2023},
  howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=mamba-selective-state-spaces}},
  note         = {AI-generated review of arXiv:2312.00752. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit — decision matrix, scorecards and trade-off analysis
Toolkit 7 decision tools Model-agnostic

Machine Learning Decision Toolkit

Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.

$24 View on Gumroad

References

Frequently asked questions

Why is Mamba linear while attention is quadratic?

Attention compares every token pair, so cost grows quadratically with length. A state-space model compresses history into a fixed-size state updated per token, so cost grows linearly — with no KV cache to store.

Does Mamba match transformers on quality?

On standard benchmarks up to its training length, yes. Very long contexts and in-context retrieval remain active research areas, and hybrid Mamba-attention models currently lead there.

How to Remain Valuable When Intelligence Becomes Cheap — book cover
eBook 224 pages

How to Remain Valuable When Intelligence Becomes Cheap

This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?

Read the eBook