Mamba: Selective State Spaces
Mamba replaces attention with selective state spaces that scale linearly in sequence length — matching transformers at several times the inference speed.
Search across AI research reports and briefs. Results load instantly — no JavaScript required.
7 results for “attention”
Mamba replaces attention with selective state spaces that scale linearly in sequence length — matching transformers at several times the inference speed.
Bahdanau attention freed neural translation from the fixed-length vector bottleneck by letting the decoder look back at the source sentence.
FlashAttention tiles exact attention in fast on-chip memory, cutting memory from quadratic to linear and making long-context transformers practical.
The Vision Transformer applied a plain transformer to image patches and matched CNNs at scale — proving convolutions were optional for recognition.
The Transformer replaced recurrence with self-attention, enabling massively parallel training — the architecture behind GPT, BERT and modern LLMs.
A synthetic proposal for causal attention: separating correlational co-occurrence from causal influence so transformers can reason over interventions.
Introduces a sequence-to-sequence model that replaces recurrence and convolution entirely with self-attention, enabling highly parallelizable training and establishing the foundation for modern large language models.