A Neural Algorithm of Artistic Style — Talk / Press / Context

Verifiable press and blog coverage

• The paper was first released on arXiv on 26 August 2015. The peer-reviewed conference version is Image Style Transfer Using Convolutional Neural Networks, published in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 2414–2423. Official Open Access page: https://openaccess.thecvf.com/content_cvpr_2016/html/Gatys_Image_Style_Transfer_CVPR_2016_paper.html
• Google AI (Google Brain) blog post Supercharging Style Transfer (October 2016) explicitly cites the paper as the foundational work and describes follow-up research by Dumoulin, Shlens and Kudlur on multi-style networks. Source: https://research.google/blog/supercharging-style-transfer/
• Wikipedia’s Neural style transfer article traces the field back to this paper and lists the original VGG-19 hyperparameters. Source: https://en.wikipedia.org/wiki/Neural_style_transfer
• The PyTorch official tutorial Neural Transfer Using PyTorch is a direct implementation of the same algorithm and cites Gatys et al. Source: https://pytorch.org/tutorials/advanced/neural_style_tutorial.html

Q&A (answers from the paper)

Q: Does style transfer retrain the neural network? A (from the paper, Methods; CVPR version Section 2): No. The VGG-19 network is already trained on ImageNet for object recognition and its weights are frozen. The only thing that changes during the optimisation is the pixel values of a new image we are synthesising. We back-propagate the content and style losses through the network all the way to those pixels.

Q: Why does a network trained to recognise objects know anything about art style? A (from the paper, Discussion): To recognise objects the network must become invariant to changes that do not affect object identity, such as lighting, colour and texture. In the process it builds layers where higher layers preserve object layout (content) and the correlations between filters at multiple layers preserve texture-like statistics (style). The paper argues this factorisation is a useful by-product of learning object recognition.

Q: Why use a Gram matrix instead of comparing raw feature maps for style? A (from the paper, Methods; CVPR version Section 2.2): Raw feature maps still contain spatial information, i.e. where objects are. The Gram matrix collapses spatial locations by computing correlations between filters, producing a texture descriptor that is largely invariant to the global arrangement of the scene.

Q: Why average pooling instead of max pooling? A (from the paper, Methods): The authors used VGG-19 but replaced the max-pooling layers with average pooling. They write that “replacing the max-pooling operation by average pooling improves the gradient flow and one obtains slightly more appealing results”.

Q: How long does the optimisation take? A (from the CVPR 2016 version, Section 4): For images of about 512 × 512 pixels, synthesis “could take up to an hour on a Nvidia K40 GPU”, depending on image size and when the optimisation is stopped. Modern GPUs are faster, but the paper’s optimisation-based approach is still slower than later feed-forward methods such as Johnson et al. (2016) or AdaIN (Huang & Belongie, 2017).

Common misconceptions

• Misconception: The network is trained to paint.
  Correction: The network is pretrained for object classification. Its internal representations are reused; only the image pixels are optimised.
• Misconception: Style is stored as a single layer.
  Correction: The paper uses a multi-scale style representation spanning conv1_1, conv2_1, conv3_1, conv4_1 and conv5_1 to capture texture at different scales.
• Misconception: Content and style can be perfectly separated.
  Correction: They cannot. The loss function trades the two objectives off; different α/β ratios give visually different results, and there is usually no image that perfectly satisfies both constraints.
• Misconception: Any CNN works equally well.
  Correction: The paper uses VGG-19 and says that replacing max pooling with average pooling gives “slightly more appealing results”. The paper reports results with VGG-19 only; it does not compare other networks.

Real citations (original paper and direct follow-ups)

• Leon A. Gatys, Alexander S. Ecker, Matthias Bethge. A Neural Algorithm of Artistic Style. arXiv:1508.06576 [cs.CV], 2015.
• Leon A. Gatys, Alexander S. Ecker, Matthias Bethge. Image Style Transfer Using Convolutional Neural Networks. CVPR 2016, pp. 2414–2423.
• Justin Johnson, Alexandre Alahi, Li Fei-Fei. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. ECCV 2016 / arXiv:1603.08155.
• Vincent Dumoulin, Jonathon Shlens, Manjunath Kudlur. A Learned Representation for Artistic Style. ICLR 2017 / arXiv:1610.07629.
• Xun Huang, Serge Belongie. Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization. ICCV 2017 / arXiv:1703.06868.