TALK — Convolutional Sequence to Sequence Learning

Press / Blog Coverage

  1. Facebook AI Research Blog (May 2017): FAIR announced the model alongside the release of fairseq, their open-source sequence-to-sequence toolkit. The blog post highlighted the speed advantages and state-of-the-art results on WMT translation benchmarks. The fairseq repository (https://github.com/facebookresearch/fairseq) was released concurrently and became one of the most-starseq NLP research tools on GitHub.
    • Repository: https://github.com/facebookresearch/fairseq
  2. Hacker News (May 2017, multiple threads): The paper generated significant community discussion, with threads debating whether convolutions would replace RNNs for sequence modeling. Many commenters noted the impressive speed gains and the elegance of the GLU gating mechanism. The concurrent release of fairseq was praised. One notable discussion point was whether this approach would scale to very long sequences or whether the fixed receptive field would become a limitation.
    • Discussed at length in threads around the arXiv submission and fairseq release.
  3. The Gradient — “Attention?” (Lilian Weng, 2018): This comprehensive survey of attention mechanisms covers ConvS2S as a key step in the evolution from recurrent attention to self-attention, noting its multi-step attention design where each decoder layer has its own attention module.
    • URL: https://lilianweng.github.io/posts/2018-06-24-attention/
  4. Sebastian Ruder’s NLP research roundups (2017): Ruder’s widely-read quarterly NLP progress reports covered this paper as a major development, noting that it was the first fully convolutional model to outperform strong RNN baselines on large-scale translation, and discussing its relationship to the concurrent Transformer work.
    • URL: https://ruder.io/
  5. Jay Alammar — “The Illustrated Transformer” (2018): While focused on the Transformer, this popular blog post contextualizes ConvS2S as part of the transition away from RNNs, noting that both ConvS2S and the Transformer addressed the parallelization bottleneck of recurrent models in different ways.
    • URL: https://jalammar.github.io/illustrated-transformer/

Interview Q&A

(Reconstructed from the paper’s content and public discussions — these are conceptual Q&As based on the paper’s own statements, not verbatim quotes from the authors.)

Q: Why use convolutions instead of RNNs for sequence modeling? The RNN + attention approach was already state-of-the-art.

A: RNNs have a fundamental limitation: they process sequences sequentially, one token at a time. This means you cannot parallelize computation across time steps, which underutilizes GPU hardware. Additionally, the number of nonlinear operations an RNN applies depends on the sequence length — the first token passes through n nonlinearities while the last passes through only one. This creates an imbalance that makes optimization harder. Convolutions solve both problems: all positions are computed simultaneously (full parallelization), and the number of nonlinearities between any input and output is fixed by the network depth, not the sequence length. This makes training faster and optimization easier.

Q: What are gated linear units (GLUs) and why are they important here?

A: A GLU takes the output of a convolution — which has twice the target dimensionality (2d) — splits it into two halves A and B (each dimension d), and computes A ⊗ σ(B), where σ is the sigmoid function and ⊗ is element-wise multiplication. The key insight is that the gradient through the sigmoid gate is always positive (sigmoid’ is bounded), which means gradients flow more easily through deep networks compared to ReLU (which can zero out gradients for negative inputs) or tanh (which saturates). In a deep convolutional stack with many layers, this ease of gradient propagation is critical for training stability.

Q: How does the multi-step attention differ from the single attention used in RNN seq2seq models?

A: In Bahdanau-style attention, there is typically one attention mechanism that computes a context vector at each decoding step, and the decoder RNN uses this context to update its hidden state. In our architecture, every decoder layer has its own separate attention module. This means attention is computed multiple times — once per layer — creating a “multi-hop” attention effect. The first decoder layer attends to the source to get an initial context, which is fed to the second layer, which can then attend to different parts of the source having already seen what the first layer focused on. This progressive refinement is richer than single-step attention and allows deeper layers to correct or build upon the attention of earlier layers.

Q: Why add the source embeddings e_j to the encoder outputs in the attention computation?

A: The encoder outputs z_j represent the context around source position j — they encode information from a window of neighboring tokens. But for attention, we sometimes need point-level information about exactly which token is at position j. Adding e_j (the raw source embedding) to z_j gives the attention both the contextual representation (from the encoder) and the point identity (from the embedding). This resembles key-value memory networks where the keys are z_j and the values are z_j + e_j. Empirically, adding e_j improved results.

Q: This paper came out just before the Transformer. How do they compare?

A: Both papers address the same fundamental problem — the sequential bottleneck of RNNs — but with different solutions. We replaced recurrence with convolutions and kept attention; the Transformer replaced both recurrence and convolutions with self-attention entirely. Our model still has a fixed receptive field determined by the kernel width and number of layers, while self-attention has an effectively global receptive field in a single layer. The Transformer achieved better results and became dominant, but our work demonstrated that parallelizable, non-recurrent architectures could reach state-of-the-art on large-scale translation — which was a necessary stepping stone. The fairseq toolkit we released for this model later became one of the primary Transformer research platforms.

Common Misconceptions

  1. “ConvS2S was the first to use convolutions for sequence modeling.” — No. Prior work included Kalchbrenner et al. (2016) with the ByteNet encoder, Bradbury et al. (2016) with quasi-recurrent neural networks, and Meng et al. (2015) with convolutional encoders for MT. Gehring et al. (2016) had also explored partially convolutional architectures. The novelty of this paper was the fully convolutional architecture with GLUs and multi-step attention that achieved state-of-the-art on large-scale translation.

  2. “The receptive field is unlimited.” — No. The receptive field is n(k-1)+1 where n is the number of layers and k is the kernel width. For 6 layers with k=5, this is 25 tokens. To handle longer dependencies, you need more layers or wider kernels. This is a fundamental difference from self-attention, which has a global receptive field in a single layer. The paper acknowledges this and uses deeper networks (up to 15 layers) to achieve sufficient receptive fields for their translation tasks.

  3. “GLUs are just another activation function like ReLU.” — While GLUs are a nonlinearity, they are fundamentally different from pointwise activations like ReLU or tanh. GLUs are a gating mechanism — they use one half of the convolution output to gate the other half, creating a learned, input-dependent nonlinearity. The gating pathway provides a gradient highway that pointwise activations cannot offer, which is why they specifically help in deep convolutional stacks.

  4. “The model cannot generate variable-length outputs.” — It can. The decoder generates one token at a time autoregressively, using an end-of-sequence token to stop generation. The convolutional structure enables parallel computation during training (when the full target sequence is known and teacher forcing is used), but inference is still autoregressive. The parallelization advantage is primarily a training-time benefit, though the paper also reports faster inference due to fewer total operations.

  5. “fairseq was created for the Transformer.” — No. fairseq was originally created for this ConvS2S model and released alongside it in May 2017. Transformer support was added later. fairseq’s name itself derives from “fully convolutional sequence-to-sequence” — “fair” from FAIR (Facebook AI Research) and “seq” from sequence. The toolkit later became one of the most widely used Transformer research platforms, but its origin is this paper.

Real Citations

  • Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017). Convolutional Sequence to Sequence Learning. Proceedings of the 34th International Conference on Machine Learning (ICML 2017). arXiv:1705.03122.
  • Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NeurIPS 2017). arXiv:1706.03762.
  • Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. NeurIPS 2014. arXiv:1409.3215.
  • Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015. arXiv:1409.0473.
  • Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., et al. (2016). Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv:1609.08144.
  • Dauphin, Y. N., Fan, A., Auli, M., & Grangier, D. (2017). Language Modeling with Gated Convolutional Networks. ICML 2017. arXiv:1612.08083.
  • Kalchbrenner, N., Espeholt, L., Simonyan, K., van den Oord, A., Graves, A., & Kavukcuoglu, K. (2016). Neural Machine Translation in Linear Time. arXiv:1610.10099.
  • Bradbury, J., Merity, S., Xiong, C., & Socher, R. (2017). Quasi-Recurrent Neural Networks. ICLR 2017. arXiv:1611.01576.
  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. CVPR 2016. arXiv:1512.03385.
  • van den Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016). Pixel Recurrent Neural Networks. ICML 2016. arXiv:1601.06759.