Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau attention freed neural translation from the fixed-length vector bottleneck by letting the decoder look back at the source sentence.
12 reports · 2014–2023
The network designs that mattered: attention, convolutions, residual and graph models, sequence-to-sequence and their descendants.
Bahdanau attention freed neural translation from the fixed-length vector bottleneck by letting the decoder look back at the source sentence.
The Sutskever sequence-to-sequence LSTM encoded sentences to a vector and decoded translations — the encoder-decoder pattern behind neural translation.
An AI-generated review of ResNet, the 2015 architecture that used skip connections to train networks hundreds of layers deep and became the default backbone.
An AI-generated review of the Graph Convolutional Network (GCN), which extended deep learning to graph-structured data with a simple layer-wise propagation.
The Transformer replaced recurrence with self-attention, enabling massively parallel training — the architecture behind GPT, BERT and modern LLMs.
BERT pre-trained deep bidirectional transformers with masked language modeling, then fine-tuned one model per task — the blueprint for modern NLP.
An AI-generated review of T5, the framework that cast every NLP task as text-to-text and systematically studied scale, data, and recipe choices for transfer.
RoBERTa showed BERT was undertrained: longer training, more data and dropped objectives substantially improved results with the same architecture.
XLNet combines permutation language modeling with Transformer-XL recurrence, outperforming BERT on SQuAD, GLUE and RACE benchmarks.
Rotary position embeddings encode token order as rotations in embedding space, giving transformers relative positions that stretch to longer text.
A synthetic proposal for causal attention: separating correlational co-occurrence from causal influence so transformers can reason over interventions.
Mamba replaces attention with selective state spaces that scale linearly in sequence length — matching transformers at several times the inference speed.
Narrated films that touch this topic, with chapters, captions and transcripts.
1:40Explainer · 4 chapters
Why a language model feels like it understands, what it is actually doing, and three questions that replace “does it understand?”.