Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau attention freed neural translation from the fixed-length vector bottleneck by letting the decoder look back at the source sentence.
11 steps · about 74 min · 2014–2024
Start with the attention mechanism, meet the original Transformer, then follow the line through BERT, GPT-3, T5, RoBERTa, vision transformers, rotary embeddings and the move away from attention with Mamba and sparse mixture-of-experts.
Prefer to jump around? Each step is a standalone report with its own citations, figures and FAQ.
Bahdanau attention freed neural translation from the fixed-length vector bottleneck by letting the decoder look back at the source sentence.
The Transformer replaced recurrence with self-attention, enabling massively parallel training — the architecture behind GPT, BERT and modern LLMs.
BERT pre-trained deep bidirectional transformers with masked language modeling, then fine-tuned one model per task — the blueprint for modern NLP.
RoBERTa showed BERT was undertrained: longer training, more data and dropped objectives substantially improved results with the same architecture.
An AI-generated review of GPT-3, the 175-billion-parameter language model that showed scale alone can unlock in-context learning without any gradient updates.
An AI-generated review of T5, the framework that cast every NLP task as text-to-text and systematically studied scale, data, and recipe choices for transfer.
The Vision Transformer applied a plain transformer to image patches and matched CNNs at scale — proving convolutions were optional for recognition.
Rotary position embeddings encode token order as rotations in embedding space, giving transformers relative positions that stretch to longer text.
Mamba replaces attention with selective state spaces that scale linearly in sequence length — matching transformers at several times the inference speed.
Mixtral routes each token to 2 of 8 experts, matching LLaMA-2 70B quality with 6x faster inference — open sparse MoE that actually works.
LLaMA showed a 13B model trained on a trillion tokens could beat GPT-3 175B, reframing the scaling debate around compute-optimal token budgets.