Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau attention freed neural translation from the fixed-length vector bottleneck by letting the decoder look back at the source sentence.
Primary source: Neural Machine Translation by Jointly Learning to Align and Translate (arXiv:1409.0473)
The first seq2seq translation models had a well-known weakness: the encoder had to squeeze the entire source sentence into a single fixed-length vector, and translation quality collapsed on long sentences. In 2014, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio proposed a mechanism that lets the decoder look back at the source — attending to the most relevant words at each step — instead of relying on one compressed summary. This attention mechanism removed the bottleneck, improved long-sentence translation, and became the single most influential idea in modern sequence modeling.
The core idea
Replace the fixed-length context vector with a dynamic context vector computed at each decoding step. At step t, the decoder:
- Computes a score between its current hidden state and every encoder hidden state (using a small learned alignment model — an additive MLP).
- Normalizes the scores into attention weights (softmax).
- Takes a weighted sum of encoder states — the context vector — and feeds it to the decoder.
The alignment is learned jointly with translation: the network discovers which source words to focus on while translating, without any explicit alignment supervision.
How it works
- Encoder: a bidirectional RNN (one forward, one backward) whose concatenated hidden states represent each source word with both left and right context.
- Decoder: an RNN that at each step computes the context vector via attention, then uses it — concatenated with the previous target embedding — to produce the next word.
- Alignment model: a small feed-forward network that scores decoder state s(t-1) against each encoder state h(j); the paper's original formulation uses a tanh MLP, later simplified to dot products in the Transformer.
- Training: end-to-end with backpropagation; the attention weights emerge without alignment annotations.
The paper evaluated on English-to-French WMT-14. The attentive model beat the fixed-vector baseline by a large margin, and — critically — the gap grew with sentence length: the baseline collapsed beyond ~30 words while the attention model stayed strong at 60+ words.
Why it matters
- Soft alignment as a learned behavior. The attention weights are inspectable: they visibly align source and target words, including non-monotonic reorderings (e.g., adjective-noun inversion between English and German).
- Long-range dependency fix. Attention let models route information directly across arbitrary distances, sidestepping the vanishing-gradient problem of sequential compression.
- The parent of the Transformer. "Attention is all you need" replaced the RNNs with pure attention — but the mechanism itself is exactly this paper's idea, generalized.
- Beyond MT. Attention became the building block for summarization, speech recognition, image captioning, and retrieval (the same softmax-weighted averaging is the core of modern transformers and cross-attention).
Limitations and open questions
The original attention has quadratic cost in sequence length — a limitation that spawned FlashAttention and sparse attention variants. RNN-based encoders remained sequential, so the paper's architecture still trained slowly; the Transformer solved parallelism but kept attention. The additive alignment model was later replaced by cheaper dot-product scoring, which the Transformer adopted with a scaling factor.
Key takeaways
- Attention = learned, differentiable alignment between decoder state and encoder states.
- It eliminated the fixed-length bottleneck and preserved quality on long sentences.
- Attention weights provide interpretable alignments without supervision.
- The mechanism is the direct ancestor of self-attention and cross-attention in modern transformers.
Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.
Cite this report
@misc{knowyourml-attention-neural-machine-translation
title = {Neural Machine Translation by Jointly Learning to Align and Translate},
author = {{Know Your ML}},
year = {2014},
howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=attention-neural-machine-translation}},
note = {AI-generated review of arXiv:1409.0473. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit
Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.
References
- Neural Machine Translation by Jointly Learning to Align and Translate (arXiv:1409.0473) — https://arxiv.org/abs/1409.0473
Part of
Beyond the paper
How to Remain Valuable When Intelligence Becomes Cheap
This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?