11 steps · about 74 min · 2014–2024

The Transformer lineage

Start with the attention mechanism, meet the original Transformer, then follow the line through BERT, GPT-3, T5, RoBERTa, vision transformers, rotary embeddings and the move away from attention with Mamba and sparse mixture-of-experts.

Fig. 02 / Reading route11 papers, in order
Reading path: Neural Machine Translation by Jointly Learning to Align and Translate; The Transformer: Attention Without Recurrence; BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding; RoBERTa: A Robustly Optimized BERT Pretraining Approach; GPT-3: Language Models are Few-Shot Learners; Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer; An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale; RoPE: Rotary Position Embeddings; Mamba: Selective State Spaces; Mixtral: Sparse Mixture of Experts; LLaMA: Open and Efficient Foundation Language Models0102030405060708091011
Start with Neural Machine Translation by Jointly Learning to Align and Translate. Follow the numbered steps to LLaMA: Open and Efficient Foundation Language Models.
  1. Neural Machine Translation by Jointly Learning to Align and Translate 2014 · 7 min read · Beginner — Bahdanau attention freed neural translation from the fixed-length vector bottleneck by letting the decoder look back at the source sentence.
  2. The Transformer: Attention Without Recurrence 2017 · 5 min read · Beginner — The Transformer replaced recurrence with self-attention, enabling massively parallel training — the architecture behind GPT, BERT and modern LLMs.
  3. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 2018 · 8 min read · Intermediate — BERT pre-trained deep bidirectional transformers with masked language modeling, then fine-tuned one model per task — the blueprint for modern NLP.
  4. RoBERTa: A Robustly Optimized BERT Pretraining Approach 2019 · 7 min read · Advanced — RoBERTa showed BERT was undertrained: longer training, more data and dropped objectives substantially improved results with the same architecture.
  5. GPT-3: Language Models are Few-Shot Learners 2020 · 9 min read · Advanced — An AI-generated review of GPT-3, the 175-billion-parameter language model that showed scale alone can unlock in-context learning without any gradient updates.
  6. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer 2019 · 9 min read · Advanced — An AI-generated review of T5, the framework that cast every NLP task as text-to-text and systematically studied scale, data, and recipe choices for transfer.
  7. An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale 2020 · 7 min read · Intermediate — The Vision Transformer applied a plain transformer to image patches and matched CNNs at scale — proving convolutions were optional for recognition.
  8. RoPE: Rotary Position Embeddings 2021 · 5 min read · Intermediate — Rotary position embeddings encode token order as rotations in embedding space, giving transformers relative positions that stretch to longer text.
  9. Mamba: Selective State Spaces 2023 · 5 min read · Advanced — Mamba replaces attention with selective state spaces that scale linearly in sequence length — matching transformers at several times the inference speed.
  10. Mixtral: Sparse Mixture of Experts 2024 · 5 min read · Intermediate — Mixtral routes each token to 2 of 8 experts, matching LLaMA-2 70B quality with 6x faster inference — open sparse MoE that actually works.
  11. LLaMA: Open and Efficient Foundation Language Models 2023 · 7 min read · Intermediate — LLaMA showed a 13B model trained on a trillion tokens could beat GPT-3 175B, reframing the scaling debate around compute-optimal token budgets.

Every step as a full report

Prefer to jump around? Each step is a standalone report with its own citations, figures and FAQ.

The Transformer: Attention Without Recurrence

Architectures · 5 min read · Beginner

The Transformer replaced recurrence with self-attention, enabling massively parallel training — the architecture behind GPT, BERT and modern LLMs.

transformersattentionself-attention Read

GPT-3: Language Models are Few-Shot Learners

Foundations & scaling · 9 min read · Advanced

An AI-generated review of GPT-3, the 175-billion-parameter language model that showed scale alone can unlock in-context learning without any gradient updates.

gpt-3llmsfew-shot-learning Read

RoPE: Rotary Position Embeddings

Architectures · 5 min read · Intermediate

Rotary position embeddings encode token order as rotations in embedding space, giving transformers relative positions that stretch to longer text.

position-embeddingsropetransformers Read

Mamba: Selective State Spaces

Architectures · 5 min read · Advanced

Mamba replaces attention with selective state spaces that scale linearly in sequence length — matching transformers at several times the inference speed.

mambastate-space-modelslong-context Read

Mixtral: Sparse Mixture of Experts

Efficiency & deployment · 5 min read · Intermediate

Mixtral routes each token to 2 of 8 experts, matching LLaMA-2 70B quality with 6x faster inference — open sparse MoE that actually works.

mixture-of-expertsmoeefficiency Read

LLaMA: Open and Efficient Foundation Language Models

Foundations & scaling · 7 min read · Intermediate

LLaMA showed a 13B model trained on a trillion tokens could beat GPT-3 175B, reframing the scaling debate around compute-optimal token budgets.

large-language-modelsscaling-lawstraining Read