AI-generated review 7 min read Beginner 515 words

Sequence to Sequence Learning with Neural Networks

The Sutskever sequence-to-sequence LSTM encoded sentences to a vector and decoded translations — the encoder-decoder pattern behind neural translation.

seq2seq lstm machine-translation encoder-decoder history
Architectures · 2014

Primary source: Sequence to Sequence Learning with Neural Networks (arXiv:1409.3215)

In 2014, most NLP systems still treated translation as a pipeline of engineered components. Ilya Sutskever, Oriol Vinyals, and Quoc Le showed that a pair of LSTMs — one reading the source sentence, one generating the target sentence — could translate end-to-end, and that reversing the source sequence dramatically improved results. The sequence-to-sequence (seq2seq) framework they introduced became the template for neural machine translation, summarization, dialogue, and ultimately every encoder-decoder generative model that followed.

The core idea

Map an input sequence (any length) to an output sequence (any length) with a single neural network:

  1. Encoder: an LSTM reads the source sentence token by token and compresses it into a fixed-length vector — the final hidden state, called the thought vector.
  2. Decoder: a second LSTM, initialized with that vector, generates the target sentence one token at a time, conditioning each prediction on the previous generated token.

The whole system is trained end-to-end to maximize the probability of the target sentence given the source sentence. No alignment, no phrase tables, no reordering rules.

How it works

  • Architecture: four-layer LSTMs (each with 1,000 units per direction), trained with stochastic gradient descent, gradient clipping, and batches of roughly 6,000 source words.
  • Two key tricks:

- Reverse the source sentence. The authors found that reversing the input (mapping a b c to c b a) reduced the average distance between corresponding words, which dramatically helped the LSTM learn long-range dependencies and improved BLEU by several points. The final model used an ensemble of five such networks.

- Reverse the target sentence at decoding time. Output was generated in reverse order as well, which let the model start from the (statistically easier) first words of the translation.

  • Result: BLEU scores of 34.8 on WMT-14 English-to-French and 30.6 English-to-German, close to the best phrase-based statistical systems of the time — with none of the linguistic engineering.

Why it matters

  • End-to-end learning for structured outputs. Seq2seq showed that sequence transduction needs no explicit structure — a claim that powered the entire neural MT revolution and later summarization, speech, and code models.
  • Encoder-decoder abstraction. The architecture's separation of "read" and "write" phases is still the shape of modern encoder-decoder transformers (T5, BART, and the original Transformer).
  • Training recipes. Reversing inputs, deep LSTMs, gradient clipping, and ensemble decoding became standard practice in the neural sequence modeling toolbox.
  • It directly motivated the attention mechanism (Bahdanau et al., 2014), which removed the fixed-length bottleneck the paper's authors had to work around.

Limitations and open questions

The fixed-length thought vector is the architecture's fundamental weakness: long sentences compress information into a single vector and degrade. The paper needed the input-reversal trick precisely because of this bottleneck. Later, attention, then transformers, eliminated the fixed vector entirely. LSTMs also processed sequences sequentially, which limited parallelism — the reason transformers eventually replaced them.

Key takeaways

  • Seq2seq = encoder LSTM → fixed vector → decoder LSTM, trained end-to-end for sequence transduction.
  • Input reversal and output reversal were surprising, effective engineering tricks that improved BLEU significantly.
  • The model matched phrase-based statistical MT without any linguistic engineering.
  • The fixed-length bottleneck inspired attention, the mechanism that led to the Transformer.

Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.

Cite this report
@misc{knowyourml-seq2seq-sequence-learning
  title        = {Sequence to Sequence Learning with Neural Networks},
  author       = {{Know Your ML}},
  year         = {2014},
  howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=seq2seq-sequence-learning}},
  note         = {AI-generated review of arXiv:1409.3215. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit — decision matrix, scorecards and trade-off analysis
Toolkit 7 decision tools Model-agnostic

Machine Learning Decision Toolkit

Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.

$24 View on Gumroad

References

How to Remain Valuable When Intelligence Becomes Cheap — book cover
eBook 224 pages

How to Remain Valuable When Intelligence Becomes Cheap

This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?

Read the eBook