Talks, Coverage & Discussion — Dynamic Routing Between Capsules

Press & Blog Coverage

  1. Google Research Blog (2017): The paper was highlighted in Google’s AI research communications as a novel approach to neural network architecture that moves beyond traditional CNNs. Sara Sabour presented at NIPS 2017.

  2. “Understanding Hinton’s Capsule Networks” — Max Pechyonkin’s blog series (2017-2018): A widely-read 5-part series that breaks down the paper’s architecture, routing algorithm, squashing function, and margin loss for practitioners. Available at https://pechyonkin.me/archive/

  3. “Capsule Networks (CapsNets) – Tutorial” — Aurélien Géron’s video tutorial (2018): A comprehensive YouTube walkthrough implementing CapsNets in TensorFlow, one of the most-watched capsule network tutorials.

  4. distill.pub and multiple ML blog discussions: CapsNets were widely discussed in the ML community as a potential successor to CNNs, with detailed visual explanations of routing-by-agreement.

  5. “What is a Capsule Network?” — Matthew Stewart (Towards Data Science, 2018): An accessible explanation of capsule networks with diagrams and implementation notes.

Interview-Style Q&A

Q1: What exactly is a “capsule” and why is it different from a regular neuron?

A: A capsule is a group of neurons whose combined output vector represents both the probability that an entity exists (vector length) and its instantiation parameters — things like pose, scale, orientation, and thickness (vector direction). A regular neuron outputs a single scalar, losing all spatial/pose information. Think of it as the difference between saying “I’m 90% sure there’s a face” versus “I’m 90% sure there’s a face, tilted 15°, with thick lips, and shifted slightly to the left.”

Q2: Why does routing-by-agreement replace max-pooling, and what’s the advantage?

A: Max-pooling is a crude routing mechanism: it picks the strongest activation in a region and discards everything else, throwing away positional information. Routing-by-agreement is input-dependent and iterative: lower-level capsules make predictions about higher-level capsules, and the coupling is strengthened when predictions agree. This means the network learns to route information dynamically based on the actual content of the image, not just local maxima. The advantage is better preservation of spatial relationships — essential for tasks like recognizing overlapping digits.

Q3: The paper shows CapsNets are robust to affine transformations. Why?

A: Because each DigitCaps capsule explicitly encodes pose in the vector orientation. When the same digit is shifted or slightly rotated, the capsule’s activity vector changes in a predictable way — the direction shifts to reflect the new pose. A traditional CNN, having thrown away spatial info via max-pooling, has to relearn this from scratch with augmented data. The paper shows a CapsNet trained only on translated MNIST achieves 79% on affNIST (affine-transformed MNIST) vs 66% for a comparable CNN.

Q4: If CapsNets are so good, why didn’t they replace CNNs?

A: Two main reasons: (1) Computational cost — the routing algorithm requires multiple iterations per forward pass, and the number of capsule connections grows quadratically. (2) Scaling difficulties — CapsNets worked well on MNIST and small datasets but struggled to match the performance of CNNs/Transformers on ImageNet-scale problems. The routing mechanism also becomes harder to train on larger, more diverse datasets. However, the ideas influenced subsequent work in equivariant networks and attention mechanisms.

Q5: What does the 16-dimensional DigitCaps vector actually encode?

A: The paper’s Figure 4 shows that when individual dimensions of the 16D vector are perturbed (tweaked by ±0.25), the reconstructed image changes in interpretable ways: some dimensions control stroke thickness, others control scale, skew, width, or localized part variations. This demonstrates that the capsules have learned to encode meaningful instantiation parameters — something a scalar-output CNN cannot explicitly show.

Common Misconceptions

  1. “CapsNets are just CNNs with vectors instead of scalars.” — Not quite. The key innovation is the routing-by-agreement mechanism, which is fundamentally different from the fixed, non-learned routing of max-pooling. The iterative agreement process is a form of dynamic, input-dependent attention.

  2. “CapsNets achieve SOTA on all tasks.” — The paper achieves SOTA on MNIST (0.25% error) and demonstrates good performance on MultiMNIST (overlapping digits), but CapsNets have not been shown to outperform CNNs or Transformers on large-scale tasks like ImageNet.

  3. “The routing algorithm is just attention.” — While routing-by-agreement shares similarities with attention mechanisms (both compute weighted combinations), routing is iterative and specifically designed for the capsule framework (measuring agreement between prediction vectors and parent outputs). The paper frames it as “explaining away” rather than general attention.

  4. “CapsNets can’t do reconstruction.” — They can, via the decoder regularizer. The reconstruction is not the primary output but a training signal that encourages capsules to encode meaningful pose information. The reconstructions are remarkably faithful, showing that the 16D vector captures rich information about the input.

Real Citations

  • Sabour, S., Frosst, N., & Hinton, G. E. (2017). “Dynamic Routing Between Capsules.” arXiv:1710.09829. Presented at NIPS 2017.
  • Hinton, G. E., Krasthev, N., & Wang, S. (2011). “Transforming auto-encoders.” In ICANN. — The original concept of capsules, referenced as foundational work.
  • Hinton, G. E., et al. (2000). “Learning to Represent Spatial Relationships.” — Early work on capsule-like ideas.
  • The paper has been cited 5,000+ times according to Google Scholar, making it one of the most influential neural architecture papers of the late 2010s.
  • Subsequent work: Hinton’s “Capsules for Object Segmentation” (2018), Xi et al. “Matrix Capsules with EM Routing” (ICLR 2018), and numerous architectural variants inspired by the routing mechanism.