TALK — Pointer Networks

Press / Blog Coverage

  1. Distill.pub — “Attention and Augmented Recurrent Neural Networks” (Olah & Carter, 2016): This widely-read visual explainer covers Pointer Networks as a key example of attention used as a pointer mechanism, with interactive visualizations. The article references Pointer Networks (Vinyals et al., 2015) directly and discusses how attention-as-pointing differs from attention-as-blending.
    • URL: https://distill.pub/2016/augmented-rnns/

  2. Google Research Blog: Oriol Vinyals was a researcher at Google, and Pointer Networks was published while he was there. The paper was presented at NeurIPS 2015 and highlighted in Google Research’s publications. The work is frequently cited in Google’s broader neural architectures and sequence modeling research outputs.

  3. Abigail See’s blog — “Get To The Point: Summarization with Pointer-Generator Networks” (2017): This influential Stanford NLP blog post and paper directly builds on Pointer Networks, explaining the pointer mechanism clearly: “At each timestep, we may either generate a word from the vocabulary, or (new) copy a word from the source via the pointer.” The post is widely referenced in NLP courses.
    • URL: https://web.archive.org/web/20210812093810/https://research.google/blog/get-to-the-point-summarization-with-pointer-generator-networks/ (original Google Research blog post)
    • Paper: https://arxiv.org/abs/1704.04368

  4. Hacker News discussions (multiple threads): The paper and its follow-up (Pointer-Generator Networks) generated significant community discussion, with debates about whether neural networks can truly learn algorithmic solutions from examples and how pointer mechanisms compare to Neural Turing Machines for combinatorial problems.

  5. The Gradient — “Attention?” (Lilian Weng, 2018): This comprehensive survey of attention mechanisms traces the evolution from Bahdanau attention through Pointer Networks to transformers, explaining how Pointer Networks repurposed attention from a blending tool to a selection tool.
    • URL: https://lilianweng.github.io/posts/2018-06-24-attention/

Interview Q&A

(Reconstructed from the paper’s content and public discussions — these are conceptual Q&As based on the paper’s own statements, not verbatim quotes from the authors.)

Q: How is the pointer mechanism different from standard seq2seq attention?

A: In standard seq2seq with attention (Bahdanau et al., 2015), the attention weights are used to compute a weighted sum of encoder hidden states — a context vector — which is concatenated with the decoder state and passed through a linear layer to produce a distribution over a fixed output vocabulary. The attention is a means to an end (better context for the output layer). In Pointer Networks, we skip the output layer entirely. The attention distribution is the output distribution — over the input positions, not over a fixed vocabulary. This is why we call it a “pointer”: it directly selects an element of the input.

Q: Why can’t regular seq2seq models solve these problems?

A: The fundamental issue is that the output dictionary size depends on the input length. For convex hull, if the input has n points, the output is a sequence of indices from 1 to n. A seq2seq model has a fixed-size softmax output — it always predicts from the same vocabulary. You would need to retrain the model for each input length, or pad to a maximum length and ignore the extra positions, which is wasteful and doesn’t generalize. Our pointer mechanism makes the output dictionary size equal to the input length automatically — it’s inherent in the attention computation.

Q: Can the model really learn to solve convex hulls from examples alone?

A: Yes, and this was one of the most surprising results. The model has no prior knowledge of geometry or convex hull algorithms. It only sees input-output pairs: here are some points, here is the correct hull. Through gradient descent on the cross-entropy loss, it learns to attend to the right points in the right order. On sequences of length 50, it achieves 98.6% accuracy. More impressively, it generalizes to sequences longer than those seen in training — a model trained on 5–50 points performs reasonably on 60–80 point inputs. This suggests the model learns a general algorithm rather than memorizing specific configurations.

Q: How does this relate to Neural Turing Machines, which also deal with external memory and addressing?

A: Both are concerned with variable-size addressing, but the approaches are quite different. NTMs use a combination of content-based and location-based addressing with differentiable read/write heads, and they operate iteratively with a controller. Pointer Networks are simpler — they use attention as a one-shot pointer at each decoding step, without iterative refinement or explicit memory writes. NTMs target algorithmic tasks like copying and sorting; we target combinatorial optimization problems like convex hull and TSP. Our approach is more directly suited for sequence-to-sequence problems where the output is a permutation or subset of the input.

Q: What are the limitations of Pointer Networks?

A: The main limitation is that the output must be a sequence of indices into the input — you can only point to things you’ve seen. There’s no way to generate a token that isn’t in the input. This is why the follow-up Pointer-Generator Network (See et al., 2017) combined the pointer mechanism with a standard vocabulary, allowing both copying from the source and generating from a vocabulary. Another limitation is that for TSP, the model achieves only near-optimal (not optimal) solutions, and performance degrades significantly for large instances — the model doesn’t scale to the hundreds or thousands of cities that traditional TSP heuristics handle.

Common Misconceptions

  1. “Pointer Networks are just seq2seq with attention.” — No. Standard seq2seq with attention (Bahdanau et al.) uses attention to compute a context vector that feeds into a fixed-vocabulary output layer. Pointer Networks replace the output layer — the attention distribution is the output distribution over input positions. There is no separate vocabulary. This is a fundamental architectural difference, not a minor variation.

  2. “The model learns the convex hull algorithm.” — Not exactly. The model learns an approximation of the convex hull algorithm from examples. It achieves high accuracy on test data but doesn’t implement the exact computational steps of, say, Graham scan or Andrew’s monotone chain. It may learn a different — possibly more holistic — strategy for identifying boundary points. The “algorithm” it learns is implicit in the network weights.

  3. “Pointer Networks can only solve geometric problems.” — No. The paper demonstrates convex hull, Delaunay triangulation, and TSP, but the pointer mechanism is general — any problem where the output is a sequence of indices into the input can be addressed. The follow-up work applied pointer mechanisms to text summarization (pointing to source words), dependency parsing (pointing to head words), and code suggestion. The pointer mechanism is task-agnostic.

  4. “The model can handle arbitrary input lengths.” — While the model generalizes to longer inputs than seen in training, this generalization is not unlimited. Performance degrades as input length increases significantly beyond the training range. The LSTM encoder still has a fixed hidden state size, which creates an information bottleneck for very long sequences. The paper shows good generalization up to ~2x the training maximum, but not to arbitrarily long inputs.

  5. “Beam search is necessary for good results.” — The paper uses beam search with beam size 5 for evaluation, which does improve results. However, the model still performs well with greedy (argmax) decoding. Beam search helps mostly for the TSP task where the ordering matters more. For convex hull, greedy decoding achieves near-identical accuracy because the hull structure is more constrained.

Real Citations

  • Vinyals, O., Fortunato, M., & Jaitly, N. (2015). Pointer Networks. Advances in Neural Information Processing Systems 28 (NeurIPS 2015). arXiv:1506.03134.
  • Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. International Conference on Learning Representations (ICLR 2015). arXiv:1409.0473.
  • Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. Advances in Neural Information Processing Systems 27 (NeurIPS 2014). arXiv:1409.3215.
  • See, A., Liu, P. J., & Manning, C. D. (2017). Get To The Point: Summarization with Pointer-Generator Networks. ACL 2017. arXiv:1704.04368.
  • Ma, X., Hu, Z., Liu, Z., Peng, N., & Neubig, G. (2018). Stack-Pointer Networks for Dependency Parsing. ACL 2018. arXiv:1805.01087.
  • Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016). Pointer Sentinel Mixture Models. arXiv:1609.07843.
  • Bello, I., Pham, H., Le, Q. V., Norouzi, M., & Bengio, S. (2016). Neural Combinatorial Optimization with Reinforcement Learning. arXiv:1611.09940.
  • Kool, W., van Hoof, H., & Welling, M. (2019). Attention, Learn to Solve Routing Problems! International Conference on Learning Representations (ICLR 2019). arXiv:1803.08475.
  • Vinyals, O., Bengio, S., & Kudlur, M. (2015). Order Matters: Sequence to sequence for sets. arXiv:1511.06391.
  • Olah, C., & Carter, S. (2016). Attention and Augmented Recurrent Neural Networks. Distill. https://distill.pub/2016/augmented-rnns/