The Transformer: Attention Without Recurrence
The Transformer replaced recurrence with self-attention, enabling massively parallel training — the architecture behind GPT, BERT and modern LLMs.
Primary source: Attention Is All You Need (arXiv:1706.03762)
Before 2017, the dominant way to model sequences — sentences, audio, time series — was with recurrent neural networks (RNNs) and their gated variants such as LSTMs and GRUs. These models read input one token at a time, carrying a hidden state forward like a memory stream. They worked, but they were hard to parallelize and struggled to connect distant tokens.
The core idea
The 2017 paper Attention Is All You Need proposed a radically simpler architecture: the Transformer. Instead of recurrence or convolution, it is built from one mechanism, called self-attention, plus small feed-forward networks.
Self-attention lets every token in a sequence look at every other token at once. For each word, the model learns three vectors:
- Query: what am I looking for?
- Key: what do I contain?
- Value: what do I contribute?
By comparing queries and keys, the model computes a weighted mix of values. This single operation captures relationships between any two positions in the input, regardless of distance.
The attention formula
Stack the queries, keys and values of all tokens into matrices Q, K and V (one row per token). The paper's scaled dot-product attention is:
Each row of QK^T scores one token against every token. Softmax turns the row into weights that sum to one, and those weights mix the value vectors. The division by \sqrt{d_k} is deliberate: for a large key dimension d_k the dot products grow large, which pushes softmax into regions where its gradient is tiny. Scaling keeps training stable.
Inside the architecture
The original model is an encoder-decoder stack. In the base configuration each side has six identical layers, the model dimension is 512, and the feed-forward sublayer has an inner dimension of 2048.
- Multi-head attention: eight attention "heads" run in parallel, each on a 64-dimensional slice, so the model can attend to different kinds of relationships at the same time. Their outputs are concatenated and projected back.
- Three ways attention is used: self-attention in the encoder; masked self-attention in the decoder, so a position cannot peek at later positions it has not generated yet; and encoder-decoder attention, where the decoder's queries read the encoder's keys and values.
- Positional encoding: self-attention by itself ignores order, so the model adds sine and cosine signals of different frequencies to the token embeddings. The authors report that learned position embeddings performed nearly the same.
- Residual connections and layer normalization: wrapped around every sublayer to keep deep stacks trainable.
- Feed-forward layers: the same small network applied to each position independently after attention.
Why it won
Removing recurrence unlocks parallelism. An RNN must process tokens one after another, so training time grows with sequence length in a way that cannot be spread across a GPU. A Transformer processes the whole sentence at once. The paper's own comparison makes the trade-off concrete:
- Self-attention: O(n^2 \cdot d) work per layer, O(1) sequential steps, and a path of length O(1) between any two positions.
- Recurrent layer: O(n \cdot d^2) work per layer, but O(n) sequential steps and a path of length O(n) between distant positions.
Short paths make long-range dependencies easier to learn, and constant sequential depth makes training far easier to parallelize. Self-attention is also cheaper per layer than recurrence whenever the sequence length n is smaller than the representation size d, which is typical for sentence-level translation.
What the paper showed
The authors tested on machine translation. The larger Transformer reached 28.4 BLEU on WMT 2014 English-to-German, more than 2 BLEU above the best earlier results including ensembles, and a single-model 41.8 BLEU on English-to-French after about 3.5 days of training on eight GPUs — a fraction of the training cost of the previous leaders. The base model trained in roughly 12 hours. They also applied it to English constituency parsing, a task very different from translation, as early evidence that the architecture was general.
Legacy
The encoder-decoder structure proved general, and then it was taken apart. Encoder-only models such as BERT, decoder-only models such as the GPT family, and encoder-decoder models such as T5 all descend from this design, and so do image models (ViT) and speech models (Whisper). The attention idea itself predates the paper: Bahdanau et al. (2014) used attention on top of a recurrent translator. The Transformer's contribution was showing that attention alone is enough.
Limitations
Self-attention compares every token with every other, so time and memory grow quadratically with sequence length. For very long inputs this becomes the bottleneck, which motivated sparse and linear attention, memory-efficient kernels such as FlashAttention, better position schemes such as rotary embeddings, and non-attention alternatives such as Mamba.
Bottom line
The Transformer replaced the sequential hidden-state idea with a direct, parallel comparison between all positions. That shift is the single most important architectural change behind the current wave of large language models.
Source
- Vaswani et al., Attention Is All You Need, 2017. arXiv:1706.03762
- Bahdanau et al., Neural Machine Translation by Jointly Learning to Align and Translate, 2014. arXiv:1409.0473
- Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018. arXiv:1810.04805
Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.
Cite this report
@misc{knowyourml-transformer-attention
title = {The Transformer: Attention Without Recurrence},
author = {{Know Your ML}},
year = {2017},
howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=transformer-attention}},
note = {AI-generated review of arXiv:1706.03762. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit
Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.
References
- Attention Is All You Need (arXiv:1706.03762) — https://arxiv.org/abs/1706.03762
- Neural Machine Translation by Jointly Learning to Align and Translate (arXiv:1409.0473) — https://arxiv.org/abs/1409.0473
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv:1810.04805) — https://arxiv.org/abs/1810.04805
Frequently asked questions
Why is the paper called "Attention Is All You Need"?
Because the model drops the two mechanisms sequence models relied on before it, recurrence and convolution, and builds everything from attention plus small feed-forward layers. The claim in the title is that attention alone is enough to model a sequence.
What is the difference between self-attention and encoder-decoder attention?
In self-attention the queries, keys and values all come from the same sequence, so tokens attend to each other. In encoder-decoder attention the queries come from the decoder while the keys and values come from the encoder output, which is how the decoder reads the source sentence.
Why does the Transformer need positional encodings?
Self-attention treats its input as an unordered set: shuffling the tokens would produce the same weights. Order information therefore has to be added to the embeddings, and the original paper does it with sine and cosine signals of different frequencies.
What is the main weakness of the Transformer?
Self-attention compares every token with every other token, so compute and memory grow quadratically with sequence length. Much of the work since 2017 is about making attention cheaper for long inputs or replacing it.
Part of
Beyond the paper
How to Remain Valuable When Intelligence Becomes Cheap
This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?