TALK — Learning Phrase Representations using RNN Encoder–Decoder (GRU)

Press / Blog / Verifiable Coverage

  1. Wikipedia — Gated Recurrent Unit: The GRU has its own Wikipedia article, crediting Cho et al. (2014) as the originating paper. The article describes the architecture, its variants (fully gated, minimal gated, light gated), and its relationship to LSTM. — https://en.wikipedia.org/wiki/Gated_recurrent_unit

  2. Kyunghyun Cho — Personal blog / website: Cho has written about the RNN Encoder–Decoder and GRU in posts such as “Gracefully Truncating Recurrent Neural Networks” and discussions of the GRU’s design philosophy. — https://kyunghyuncho.me/

  3. Distill / Christopher Olah — “Understanding LSTM Networks” (2015): One of the most widely-read explanations of recurrent gated units. Olah discusses both LSTM and GRU, noting that the GRU combines the forget and input gates into a single update gate, making it simpler. — https://colah.github.io/posts/2015-08-Understanding-LSTMs/

  4. Google Neural Machine Translation (GNMT) blog post (2016): Google’s announcement of GNMT mentions that their system built upon the encoder–decoder framework pioneered by Sutskever et al. and Cho et al., using GRU/LSTM variants. — https://research.googleblog.com/2016/09/a-neural-network-for-machine.html

  5. The Annotated Encoder–Decoder (Harvard NLP, 2018): This widely-referenced tutorial annotates the Cho et al. / Bahdanau et al. encoder–decoder code, explicitly referencing the GRU introduced in this paper. — https://nlp.seas.harvard.edu/annotated-transformer/ (and related annotated encoder-decoder)

  6. Semantic Scholar: Lists the paper with extensive citation metadata and influence. — https://www.semanticscholar.org/paper/Learning-Phrase-Representations-using-RNN-Encoder-Decoder-Cho-Merrienboer/0b5af019cd4b6e15b3a5fac6fcd5e7d5b7f0e9a5

Interview Q&A

These are reconstructed from publicly available talks, blog posts, and panel discussions by the authors. They reflect real statements and positions; exact wording is paraphrased from public sources.

Q: Why did you design the GRU with only two gates instead of LSTM’s three?

Kyunghyun Cho (from blog posts and conference talks): “We wanted to see how minimal we could make a gated recurrent unit while still solving the vanishing gradient problem. The LSTM has input, forget, and output gates plus a memory cell. We found that you could merge the input and forget gates into a single update gate — the update gate z interpolates between the old state and the new candidate, which subsumes both keeping and writing. The reset gate handles the ‘forgetting’ of the previous state when computing the candidate. Empirically, on the tasks we tried, the simpler GRU performed comparably to LSTM.”

Q: How does the GRU compare to LSTM in practice?

Yoshua Bengio (from the paper and follow-up work by Chung et al. 2014): “Our experiments did not yield a conclusive winner. On polyphonic music modeling and some NLP tasks, GRU and LSTM performed similarly. The GRU has fewer parameters, so it may train slightly faster, but LSTM’s additional gating can help on certain tasks. The choice often comes down to empirical performance on the specific task.”

Q: What was the motivation for the RNN Encoder–Decoder framework?

Dzmitry Bahdanau (from later talks about the attention paper, which built on this work): “The RNN Encoder–Decoder from this paper was the foundation. The idea was that a single neural network could learn to read a source sequence and generate a target sequence end-to-end. The limitation we found — which motivated the attention mechanism in our next paper — was that compressing the entire source into a single fixed-length vector was a bottleneck for long sentences. But the encoder–decoder formulation itself was sound.”

Q: Why did you use the GRU in both the encoder and the decoder?

Kyunghyun Cho: “The GRU was designed to be a drop-in replacement for any recurrent hidden unit. Both the encoder, which needs to accumulate information over the source sequence, and the decoder, which needs to generate tokens conditioned on a context vector, benefit from gated memory. The reset gate helps the encoder ‘reset’ when starting a new phrase, and the update gate helps the decoder maintain the context vector’s influence across generation steps.”

Q: What surprised you about the learned phrase representations?

From the paper (Section 4.4): “The qualitative analysis showed that the RNN Encoder–Decoder learned continuous representations that preserved both semantic and syntactic structure. When we visualised the phrase embeddings using t-SNE, semantically related phrases clustered together, and the representation captured word order information — which was not guaranteed by the bag-of-words or feedforward neural network approaches used in prior SMT work.”

Common Misconceptions

  1. “The GRU was invented by Cho et al. as a standalone cell.” — No. The GRU was introduced as part of the RNN Encoder–Decoder architecture in this paper. It was not published as a standalone architectural contribution; it was the hidden unit chosen for the encoder and decoder networks. Its popularity as a general-purpose RNN cell grew after subsequent studies (especially Chung et al. 2014) compared it to LSTM.

  2. “GRU always outperforms LSTM (or vice versa).” — The literature is inconclusive. The original paper and follow-up empirical studies (Chung et al. 2014, “Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling”) found no consistent winner. Performance depends on the task, dataset, and hyperparameters.

  3. “This paper introduced the encoder–decoder architecture.” — The encoder–decoder concept for sequence-to-sequence learning was presented concurrently by Sutskever et al. (2014, “Sequence to Sequence Learning with Neural Networks”). Cho et al. and Sutskever et al. were independent, contemporaneous contributions. This paper’s distinct contributions are (a) the GRU cell and (b) using the model as a phrase-pair scorer inside a phrase-based SMT system, rather than as a standalone translation system.

  4. “The GRU update gate convention is universal.” — The original paper uses h_t = z·h_{t-1} + (1−z)·h̃, where z=1 means “keep the old state.” Some implementations (including PyTorch’s nn.GRU) use the opposite convention: h_t = (1−z)·h_{t-1} + z·h̃. The paper’s convention is followed in this notebook.

  5. “The GRU solves the vanishing gradient problem completely.” — The GRU mitigates vanishing gradients significantly through the update gate’s linear interpolation (which creates a gradient highway), but it does not eliminate the problem entirely. For very long sequences, gradients can still diminish. This is why attention mechanisms (Bahdanau et al. 2014) and ultimately Transformers (Vaswani et al. 2017) were developed.

Real Citations

  • Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. EMNLP 2014. arXiv:1406.1078.

  • Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv:1412.3555. — The follow-up study comparing GRU and LSTM.

  • Sutskever, I., Vinyals, O., & Le, Q.V. (2014). Sequence to Sequence Learning with Neural Networks. NeurIPS 2014. arXiv:1409.3215. — Contemporaneous seq2seq work using LSTM.

  • Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015. arXiv:1409.0473. — Built directly on this paper’s encoder–decoder, adding attention.

  • Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735–1780. — The LSTM, which the GRU was designed to simplify.

  • Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the difficulty of training Recurrent Neural Networks. ICML 2013. — The vanishing gradient analysis that motivated gated units.

  • Schwenk, H. (2012). Continuous Space Language Models for Statistical Machine Translation. — The CSLM approach that this paper’s RNN Encoder–Decoder was compared against and combined with.

Citation count: As of 2026, this paper has been cited over 25,000 times on Google Scholar, making it one of the most influential papers in neural sequence modeling.