Talk / Press / Coverage — Sequence to Sequence Learning with Neural Networks

Verifiable press and blog coverage

  • Google Research blog, “A Neural Network for Machine Translation, at Production Scale” (27 Sep 2016).
    Authors Quoc V. Le and Mike Schuster describe how Google Translate moved from phrase-based systems to neural machine translation. The post explicitly cites this paper as the source of the direct input-sequence-to-output-sequence RNN approach: “we started using Recurrent Neural Networks (RNNs) to directly learn the mapping between an input sequence … to an output sequence [2]” where reference [2] is Sutskever, Vinyals & Le (2014).
    URL: https://blog.research.google/2016/09/a-neural-network-for-machine.html

  • Google Research blog, “Introducing tf-seq2seq: An Open Source Sequence-to-Sequence Framework in TensorFlow” (11 Apr 2017).
    The TensorFlow team released a production seq2seq framework and lists this paper as the foundational reference: “Sequence to Sequence Learning with Neural Networks, Ilya Sutskever, Oriol Vinyals, Quoc V. Le. NIPS, 2014”.
    URL: https://ai.googleblog.com/2017/04/introducing-tf-seq2seq-open-source.html

  • Google Research publications page for the paper.
    The official abstract and metadata are hosted at https://research.google/pubs/pub43155/, confirming the NeurIPS 2014 venue and the WMT’14 BLEU 34.8 result.

  • Wikipedia entry for Seq2seq.
    The article identifies Sutskever et al. (2014) as one of the two papers most commonly cited as originators of seq2seq, and notes that the research “allowed Google to overhaul Google Translate into Google Neural Machine Translation in 2016.”
    URL: https://en.wikipedia.org/wiki/Seq2seq

  • PyTorch “NLP from Scratch: Translation with a Sequence to Sequence Network and Attention” tutorial.
    Still teaches the encoder/decoder LSTM pattern introduced by the paper (and later extended with Bahdanau attention).
    URL: https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

  • OpenAlex bibliographic record.
    Shows publication year 2014, DOI 10.48550/arxiv.1409.3215, and more than 13,000 citations as of 2026.
    URL: https://openalex.org/works/W2130942839

Interview-style Q&A

Q: What was the main limitation the paper tackled?
A: In 2014, deep neural networks were excellent for fixed-size inputs and outputs (image classification, speech frames), but they could not naturally handle sequences whose lengths differed between input and output. Translation is exactly such a problem. The paper solved it by chaining an encoder LSTM with a decoder LSTM.

Q: Why LSTMs instead of plain RNNs?
A: Plain RNNs are hard to train on long sequences because of vanishing gradients. LSTMs maintain a separate cell state with gating, so they can remember information across many time steps. The paper explicitly says the standard RNN failed on the non-reversed translation task, while the LSTM succeeded.

Q: What is the “reversing trick” and why does it help?
A: The source sentence is fed to the encoder backwards. The first word of the reversed source is then physically close to the first word of the target during decoding, creating short-term dependencies that make optimization easier. The authors called the improvement surprising and substantial.

Q: Did the paper use attention?
A: No. Attention came the following year (Bahdanau et al., ICLR 2015; also the contemporaneous “RNNsearch” work). The 2014 paper encoded the whole input into a single fixed-size vector—the “bottleneck” that attention later relaxed.

Q: Is this the same as the Transformer?
A: No. Transformers (Vaswani et al., 2017) replaced recurrence with self-attention. Seq2seq LSTMs are the direct ancestor: the encoder/decoder split and the autoregressive generation recipe are the same, but the mechanism inside each block changed from recurrence to attention.

Common misconceptions

  • “Seq2seq is just machine translation.” It was introduced for translation, but the encoder/decoder framework is domain-agnostic. It has been applied to summarization, speech recognition, image captioning, code generation and chatbots.
  • “The fixed vector is a weakness, so the paper is outdated.” The fixed vector was indeed a bottleneck, but the paper proved that a single vector could carry enough information for strong translation results. That proof motivated the later attention and Transformer improvements.
  • “Reversing the source is a hack with no principled reason.” The authors explain it clearly: reversing creates shorter-range dependencies between related source and target positions, which improves gradient flow during training. It is a data-ordering insight, not a random trick.

Real citations

  • Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems, 27. arXiv:1409.3215.
  • Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv:1406.1078.
  • Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. ICLR. arXiv:1409.0473.
  • Wu, Y., et al. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144.
  • Vaswani, A., et al. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30. arXiv:1706.03762.

Note on invented content

No interview quotes or press headlines were fabricated. All coverage references above are real, verifiable URLs. The Q&A answers are derived from the paper and the cited sources, not attributed to named individuals.