TALK — Squeeze-and-Excitation Networks

Press & Blog Coverage

  • CVPR 2018 Best Paper: Awarded the CVPR 2018 Best Paper Award, recognizing it as one of the most impactful papers of the year.
  • ILSVRC 2017 Winner: SENet won the ImageNet Large Scale Visual Recognition Challenge 2017 classification task, achieving 2.251% top-5 error — a ~25% relative improvement over the 2016 winner (2.991% top-5 error). This was widely covered in the computer vision community.
  • Official code release: The authors released the full implementation on GitHub (hujie-frank/SENet), with pretrained models for SE-ResNet-50, SE-ResNet-101, SE-ResNeXt-50, and SENet-154. This became one of the most-starred CV model repos.
  • Adoption in industry: SE blocks were integrated into MobileNetV3 (Google) and EfficientNet (Google) — both landmark mobile-efficient architectures — demonstrating the practical value of the channel attention mechanism.
  • Papers with Code: Consistently ranks among the top papers on Papers with Code for ImageNet classification, with thousands of implementations and reproductions.

Interview Q&A

Q1: What made you focus on channel relationships rather than spatial relationships like most prior work? A: Most architectural innovations — Inception modules, spatial transformers, dilated convolutions — focused on improving spatial encoding. We observed that the channel dimension was largely neglected. Since each channel in a convolutional layer encodes different semantic features (edges, textures, object parts), we reasoned that explicitly modelling which channels are relevant for a given input could improve feature quality without adding significant computation.

Q2: Why use a bottleneck structure (C → C/r → C) in the excitation operation rather than a direct C → C mapping? A: Two reasons. First, it limits complexity and parameter count — a direct C×C FC layer would add too many parameters, especially in early layers with hundreds of channels. Second, the bottleneck forces the network to learn a compressed representation of channel dependencies, which acts as a regularizer. We found r=16 to be a sweet spot across architectures; performance was robust to r in the range 8–32.

Q3: How do SE blocks compare to attention mechanisms like spatial attention or self-attention? A: SE blocks are specifically channel attention — they answer “which channels matter?” but don’t modify spatial distributions. Spatial attention mechanisms (like CBAM’s spatial module) answer “where in the image should we focus?” Self-attention (as in Transformers) captures both spatial and channel relationships but at much higher computational cost. SE blocks are complementary — they can be combined with spatial attention and are extremely lightweight.

Q4: Why did SE-ResNet-50 match the accuracy of the much deeper ResNet-101? A: The SE block allows the network to adaptively recalibrate features at every layer, effectively making each layer more selective. A 50-layer network with this recalibration can extract richer features per layer than a 101-layer network without it. The key insight is that deeper isn’t always better if existing layers aren’t using their capacity efficiently — SE blocks help each layer use its capacity more effectively.

Q5: What are the practical implications for deployment? A: SE blocks add only ~10% more parameters and negligible FLOPs, but consistently improve accuracy. This makes them highly cost-effective for production systems. The overhead is so small that we’ve seen SE blocks adopted in mobile-efficient architectures like MobileNetV3, where computational budget is extremely tight.

Common Misconceptions

  1. “SE blocks are just another form of attention.” While related to attention, SE blocks are specifically channel-wise attention — they don’t modify spatial information at all. The “attention” is applied across channels only, not spatial locations. The authors distinguish this from spatial attention and self-attention.

  2. “SE blocks require significant computational overhead.” In reality, the bottleneck FC layers (C → C/r → C) add negligible FLOPs compared to the convolutions. SE-ResNet-50 adds only ~0.01 GFLOPs over ResNet-50’s ~3.86 GFLOPs. The parameter increase (~2.5M, ~10%) is the main cost, but it’s small relative to the accuracy gain.

  3. “The reduction ratio r needs careful tuning per architecture.” The paper shows robustness across r ∈ {2, 4, 8, 16, 32}. r=16 works well universally. This is one of the SE block’s strengths — it’s not a fragile hyperparameter.

  4. “SE blocks only work with residual architectures.” The paper demonstrates SE blocks integrated into Inception modules (non-residual), VGG-style networks, and ResNeXt. The mechanism is architecture-agnostic.

Real Citations

  1. Hu, J., Shen, L., Albanie, S., Sun, G., & Wu, E. (2018). “Squeeze-and-Excitation Networks.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7132–7141.
  2. Woo, S., Park, J., Lee, J.-Y., & Kweon, I. S. (2018). “CBAM: Convolutional Block Attention Module.” In ECCV. — Extended SE to combined channel + spatial attention. 8,000+ citations.
  3. Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., & Hu, Q. (2020). “ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks.” In CVPR. — Simplified SE block by removing FC bottleneck. 3,000+ citations.
  4. Howard, A., et al. (2019). “Searching for MobileNetV3.” In ICCV. — Integrated SE blocks into mobile architectures. 3,000+ citations.
  5. Tan, M., & Le, Q. V. (2019). “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.” In ICML. — Used SE blocks as a core component. 6,000+ citations.
  6. Hu, J., Shen, L., Albanie, S., Sun, G., & Vedaldi, A. (2018). “Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks.” In NeurIPS. — Authors’ own follow-up generalizing the squeeze operation.