Convolutional Sequence to Sequence Learning

Summary

Convolutional Sequence to Sequence Learning, introduced by Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin at Facebook AI Research (FAIR) in 2017, proposes a sequence-to-sequence architecture built entirely from convolutional neural networks — replacing the recurrent layers (LSTMs/GRUs) that had dominated encoder-decoder models since Sutskever et al. (2014) and Bahdanau et al. (2015). The key innovation is a fully convolutional encoder-decoder that uses gated linear units (GLUs) for nonlinearity, residual connections for depth, position embeddings for order awareness, and a separate multi-step attention module at every decoder layer. Because convolutions process all positions simultaneously (unlike RNNs that must process tokens sequentially), training is fully parallelizable across the input sequence, yielding an order-of-magnitude speedup on both GPU and CPU. The model outperformed the strong deep LSTM setup of Wu et al. (2016) on WMT’14 English-to-German and English-to-French translation while training and inferencing significantly faster. This was the last major non-Transformer sequence-to-sequence architecture to achieve state-of-the-art results on large-scale machine translation, published just one month before “Attention Is All You Need” (Vaswani et al., 2017), which would go on to dominate the field with a fully attention-based approach.

arXiv link: https://arxiv.org/abs/1705.03122

Core Idea

The central insight is that recurrent networks have a fundamental bottleneck for sequence modeling: each hidden state depends on the previous one, so you cannot parallelize computation across time steps. Convolutional networks avoid this — a 1D convolution over a sequence computes all positions simultaneously. By stacking convolutional layers with kernel width k, the receptive field grows linearly with depth: n layers with kernel k give a receptive field of n(k-1)+1, allowing the model to capture long-range dependencies in O(n/k) nonlinear operations rather than O(n) as in RNNs. This fixed, bounded number of nonlinearities also makes optimization easier — the gradient path from any output back to any input passes through a constant number of nonlinear transformations, regardless of sequence length.

The architecture pairs this with gated linear units (GLUs), which split the convolution output into two halves A and B (each of dimension d, from a 2d-dim convolution) and compute v([A B]) = A ⊗ σ(B). The sigmoid gate σ(B) controls which parts of A (the raw convolution features) pass through, easing gradient flow compared to standard nonlinearities like ReLU or tanh. Residual connections wrap each convolutional block so deep networks can be trained. Position embeddings are added to token embeddings to give the model a sense of order — critical because convolutions alone are order-equivariant. Each decoder layer has its own attention mechanism that computes a weighted sum over the final encoder layer’s outputs plus the input embeddings, providing “multi-hop” attention where deeper layers can refine what earlier layers attended to.

Key Method Details

Position embeddings: Input tokens are embedded as w_j ∈ R^f and added to learned position embeddings p_j ∈ R^f to form e_j = w_j + p_j. Same for decoder outputs.
Convolutional block structure: Each block applies a 1D convolution with kernel width k that maps k input elements (each of dimension d) to one output element of dimension 2d, followed by a GLU that halves it back to d. Residual connections add the block input to its output. Encoder pads both sides; decoder pads only the left and trims the right to prevent seeing future tokens (causal convolution).
Gated linear units (GLU): v([A B]) = A ⊗ σ(B), where the convolution output [A B] ∈ R^{2d} is split into two d-dimensional halves. The gate σ(B) selectively passes or suppresses features in A.
Multi-step attention: Each decoder layer l computes its own attention. The decoder state h_i^l is combined with the previous target embedding g_i via a linear map to produce d_i^l. Attention scores are dot-products between d_i^l and the last encoder layer’s outputs z_j^u, normalized via softmax. The conditional input c_i^l = Σ_j a_{ij}^l (z_j^u + e_j) adds both the encoder context and the source embeddings (key-value structure), and is added to the decoder output h_i^l.
Normalization: Residual block outputs are scaled by √0.5 to halve variance. Attention outputs are scaled by √(1/m) × m to counteract variance change from the weighted sum over m source elements. Encoder gradients are scaled by 1/(number of attention layers).
Initialization: Embeddings from N(0, 0.1). GLU-preceding layers from N(0, √(4/n_l)). Non-GLU layers from N(0, √(1/n_l)). Dropout-adjusted initialization accounts for variance scaling.
Results: WMT’16 English-Romanian: new SOTA, +1.9 BLEU over previous best. WMT’14 English-German: +0.5 BLEU over Wu et al. (2016). WMT’14 English-French: +1.6 BLEU. Order-of-magnitude faster training and inference on GPU and CPU.
Source code: Released as fairseq (https://github.com/facebookresearch/fairseq), which became one of the most widely used sequence modeling toolkits.

Influence

This paper was the capstone of the convolutional sequence modeling line of work (Kalchbrenner et al., 2016; Bradbury et al., 2016) and demonstrated that CNNs could match or exceed RNNs on large-scale sequence-to-sequence tasks. The fairseq toolkit, originally built for this model, became a foundational NLP research tool, later incorporating Transformer support and remaining widely used. The GLU activation found broad application beyond sequence models. The idea that parallelizable architectures could replace recurrent ones for sequence modeling was directly validated here — setting the stage for the Transformer (Vaswani et al., 2017), which appeared one month later and fully replaced both convolutions and recurrence with self-attention. The paper has been cited over 2,500 times and is a key reference in the transition from recurrent to attention-based sequence architectures.

What Problem Does It Solve

Imagine you’re translating a long sentence from English to French — say 30 words. The old way of doing this with AI was like reading the sentence one word at a time, left to right, and keeping a running “memory” of everything you’ve read so far. To translate the 30th word, the AI had to process word 1, then word 2, then word 3… all the way to word 30, one after another. You couldn’t skip ahead or work on multiple words at the same time — it was like a single worker building a wall brick by brick, one at a time. This was slow, especially on powerful computers that are designed to do many things at once. This paper showed that instead of reading one word at a time, you can use a “sliding window” approach — like looking at 5 words at once, then sliding the window over by one word and looking at the next 5. Because each window is independent, a computer can process all the windows at the same time (like hiring 30 workers instead of 1). By stacking several layers of these windows, the AI can still “see” the whole sentence — it just does it in parallel rather than one-by-one. The result: translations that are just as good (or better) but completed 10x faster, because the computer’s GPU — which is built for parallel work — is finally being used efficiently.

Resources

Kaggle Notebook:

File	Description
README.md	This overview — a 5-minute read covering the paper’s summary, core idea, key method details, influence, and a plain-English explanation of the problem it solves
solution.ipynb	Complete from-scratch PyTorch implementation of the paper’s architecture, with training loops, visualizations, and worked examples
CODE_ARCHITECTURE.md	Section-by-section breakdown of the notebook — what each cell does, data flow, tensor shapes, and key design decisions
TALK.md	Press coverage, blog posts, interview Q&A, citations, and common misconceptions about the paper