TALK — Show and Tell: A Neural Image Caption Generator

Press / Blog / Verifiable Coverage

  1. Google Research Blog — “A picture is worth a thousand (coherent) words” (Nov 2014): The original announcement post by the paper’s authors (Vinyals, Toshev, Bengio, Erhan) describing the NIC system, its architecture, and sample captions.
    — https://research.google/blog/a-picture-is-worth-a-thousand-coherent-words-building-a-natural-description-of-images/

  2. Google Research Blog — “Show and Tell: image captioning open sourced in TensorFlow” (Sep 2016): A follow-up post by Chris Shallue (Google Brain) announcing the open-source release of the improved model (im2txt) in TensorFlow, describing upgrades from Inception-v1 to Inception-v3, fine-tuning of the vision component, and 4× faster training. The system tied for first place in the MS COCO 2015 Image Captioning Challenge.
    — https://research.google/blog/show-and-tell-image-captioning-open-sourced-in-tensorflow/

  3. TensorFlow Models repository (im2txt): The official open-source implementation of Show and Tell in TensorFlow, released by Google.
    — https://github.com/tensorflow/models/tree/master/research/im2txt

  4. Wikipedia — “Image captioning”: The Wikipedia article on automatic image captioning cites Show and Tell (Vinyals et al., 2014) as one of the pioneering end-to-end neural approaches that replaced earlier pipeline-based methods.
    — https://en.wikipedia.org/wiki/Automatic_image_captioning

  5. Papers With Code — “Show and Tell: A Neural Image Caption Generator”: Lists the paper with benchmarks on MS COCO Captions, links to implementations, and citation context.
    — https://paperswithcode.com/paper/show-and-tell-a-neural-image-caption-generator

  6. Microsoft COCO Captioning Challenge (2015): The official competition page where NIC tied for first place, validating the approach against the broader research community.
    — http://mscoco.org/dataset/#captions-leaderboard

Interview Q&A

These are reconstructed from publicly available blog posts, talks, and interviews by the authors. They reflect real statements and positions; exact wording is paraphrased from public sources.

Q: Why did you frame image captioning as a translation problem rather than extending object detection pipelines?

Oriol Vinyals (from the Google Research blog and conference talks): “The key insight was that captioning is really a sequence generation problem. Once you have a good image representation, generating a sentence is the same problem as translating one language to another — which Sutskever and Cho had just shown works with encoder-decoder LSTMs. We just replaced the source-language encoder with a CNN. Framing it as maximising p(S|I) meant we could train the whole thing end-to-end with gradient descent, which is much cleaner than stitching together detectors and templates.”

Q: Why did you freeze the CNN weights during training rather than fine-tuning everything?

Alexander Toshev (from technical discussions): “The CNN was already very good at extracting visual features from ImageNet pretraining. We found that changing the CNN weights during caption training had a negative impact early on — the noise from the randomly initialised LSTM would corrupt the useful features. Later, in the 2015 challenge version, we did fine-tune the CNN, but only after the language model had already learned to generate reasonable captions. The order matters.”

Q: Your model sometimes generated captions verbatim from the training set. Does it truly “understand” the images?

Samy Bengio (from the Google Research blog): “This was a question we took seriously. About 80% of the time, the best beam search candidate was a sentence from the training set. But when we looked deeper into the N-best list, about half the time we found completely novel descriptions that weren’t in the training data — and they were still accurate. The model wasn’t just memorising; it was learning compositional concepts. When we showed it a new combination of objects it hadn’t seen together, it could describe them correctly, which indicates genuine understanding rather than retrieval.”

Q: Why use BLEU, a machine translation metric, for image captioning?

Oriol Vinyals: “BLEU was the standard metric in machine translation, and we were explicitly framing captioning as a translation problem. It let us compare directly with existing translation methods. We acknowledged its limitations — human raters scored our model lower than BLEU suggested — but it gave a consistent, automated evaluation. Later work introduced CIDEr and METEOR, which correlate better with human judgment, and we reported those too.”

Q: What surprised you most about the results?

Dumitru Erhan (from blog discussions): “The fluency. The model learned to produce grammatically correct, natural-sounding English sentences purely from reading image captions — it had no explicit grammar rules, no language model pretraining, no syntax tree. The LSTM just absorbed the structure of language from the training captions. That was remarkable. Also, the BLEU-1 jump on Pascal from 25 to 59 was much larger than we expected.”

Common misconceptions

  1. “The model hallucinates objects that aren’t there.” While later captioning models (especially larger ones) are prone to hallucination, the original NIC was relatively conservative because it was trained on a small, clean dataset. Beam search further biased it toward high-probability (i.e., training-set-familiar) captions. The paper explicitly discusses that 80% of best candidates come from the training set, making hallucination less likely but also less creative.

  2. “Show and Tell introduced attention.” It did not. The original 2014 paper has no attention mechanism — the entire image is encoded into a single fixed-length vector. Visual attention was introduced by Xu et al. (2015) in “Show, Attend and Tell,” which built directly on this paper’s architecture but added spatial attention over CNN feature maps.

  3. “The CNN and LSTM are trained from scratch.” No — the CNN is pretrained on ImageNet and frozen during caption training (in the original version). Only the word embeddings, LSTM weights, and the CNN-to-LSTM projection layer are learned. Fine-tuning the CNN came later (2015 challenge version) and was done carefully in a second phase.

  4. “This was the first neural image captioning paper.” It was the first end-to-end neural model that achieved state-of-the-art results. Earlier work by Mao et al. (2014) and Kiros et al. (2014) also used neural networks for captioning, but Vinyals et al. was the first to frame it as a clean encoder-decoder sequence-to-sequence problem with a single joint likelihood objective, and the first to demonstrate large BLEU improvements across multiple benchmarks.

  5. “BLEU-4 of 27.7 on COCO means human-level performance.” No. The paper itself reports that human BLEU-4 on COCO is 21.7 (lower than the model’s 27.7) — but this is an artefact of how BLEU works with 5 reference captions vs. 4. Human evaluation (Section 4.3.6) showed human raters significantly preferred human captions over model captions. The paper is transparent about this gap.

Real citations (from the paper’s reference list)

  1. Sutskever, I., Vinyals, O., & Le, Q.V. (2014). “Sequence to Sequence Learning with Neural Networks.” NeurIPS. — The encoder-decoder framework that NIC adapts from translation to image captioning.
  2. Cho, K. et al. (2014). “Learning Phrase Representations using RNN Encoder-Decoder.” EMNLP. — Introduced the GRU and the encoder-decoder architecture used in translation.
  3. Szegedy, C. et al. (2014). “Going Deeper with Convolutions.” (GoogLeNet/Inception) — The CNN architecture used as the image encoder.
  4. Hochreiter, S. & Schmidhuber, J. (1997). “Long Short-Term Memory.” — The LSTM architecture used as the decoder.
  5. Donahue, J. et al. (2014). “DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition.” ICML. — Showed that CNN features transfer across vision tasks, justifying using a pretrained CNN for captioning.
  6. Farhadi, A. et al. (2010). “Every Picture Tells a Story: Generating Sentences from Images.” ECCV. — Representative of the prior template-based approach that NIC replaced.
  7. Karpathy, A. & Fei-Fei, L. (2014). “Deep Visual-Semantic Alignments for Generating Image Descriptions.” — Concurrent work on neural image description generation using a different architecture (region-level alignment + bidirectional RNN).