Faster R-CNN — Talks, Coverage, and Discussion

Press / Blog Coverage

  • Papers With Code: Listed on Papers With Code for PASCAL VOC 2007, 2012, and MS COCO object detection benchmarks. One of the highest-ranked two-stage detection methods. (https://paperswithcode.com/paper/faster-r-cnn-towards-real-time-object-detection)
  • Official code repositories: The original MATLAB implementation (https://github.com/ShaoqingRen/faster_rcnn) and the popular Python/Caffe port (https://github.com/rbgirshick/py-faster-rcnn) by Ross Girshick, widely used as research baselines.
  • Detectron / Detectron2: Facebook AI Research’s detection framework (https://github.com/facebookresearch/detectron2) includes Faster R-CNN as a core model and references this paper as the foundational two-stage approach.
  • PyTorch torchvision: Faster R-CNN is included as a pretrained model in torchvision (https://pytorch.org/vision/stable/models/faster_rcnn.html), making it accessible to millions of practitioners.
  • Tryolabs blog: “Object Detection with Deep Learning: A Review” covers Faster R-CNN as a key milestone in the evolution from R-CNN → Fast R-CNN → Faster R-CNN.
  • CVPR / ICCV tutorials: Multiple computer vision summer schools and tutorials include Faster R-CNN in their detection curricula, including the ICCV 2015 LNCS tutorial by the authors.

Interview-Style Q&A

Q1: What was the main motivation for creating the Region Proposal Network?

A: After Fast R-CNN sped up the detection network to run in under 0.5 seconds, the region proposal step using Selective Search became the bottleneck — it took over 2 seconds per image on CPU. We wanted to eliminate this external dependency by making proposal generation a convolutional computation that shares features with the detector, essentially making proposals “nearly cost-free.”

Q2: How did the anchor mechanism come about?

A: We needed a way for the RPN to predict boxes of multiple scales and aspect ratios from a single feature-map location. Instead of predicting box coordinates directly, we introduced reference boxes (anchors) and had the network predict offsets relative to them. This parameterization made the regression problem much more stable. We used 3 scales and 3 aspect ratios = 9 anchors per location, which gave enough coverage without being computationally expensive.

Q3: Why share convolutional features between RPN and the detection network?

A: Both tasks — proposing regions and detecting objects — benefit from the same visual features. Sharing features means the expensive convolutional computation happens only once per image. We achieved this through alternating training: first train RPN, then train Fast R-CNN using the RPN proposals, then fine-tune both jointly. This sharing is what makes the system “Faster” — the proposal step adds only ~10ms overhead.

Q4: How did you handle the different training requirements of RPN and detection?

A: We explored two approaches: alternating training (train RPN and detector in alternating rounds) and approximate joint training (train both in one pass, treating proposals as fixed). The 4-step alternating training gave the best results. The key insight was that RPN and detector have complementary objectives — RPN learns “where are objects” while the detector learns “what are they” — but they reinforce each other through shared features.

Q5: What surprised you most about the results?

A: That the learned RPN produced higher-quality proposals than Selective Search while being orders of magnitude faster. With only 300 proposals, we matched the accuracy of using 2,000 Selective Search proposals. This showed that learned proposals are both faster and better than hand-crafted heuristics. It also validated the anchor-based approach as a principled way to handle multi-scale detection.

Common Misconceptions

  1. “Faster R-CNN is an end-to-end single network” — Not exactly. While they share convolutional features, RPN and Fast R-CNN have separate loss functions and can be trained in alternating or approximate-joint mode. True end-to-end joint training with a single combined loss is an approximation used in practice but the paper’s best results used 4-step alternating training.

  2. “Anchors are the same as default boxes in SSD” — While conceptually similar, anchors in Faster R-CNN are used by the RPN to generate proposals (a separate proposal stage), while SSD’s default boxes are used directly for final detection in a single stage. Anchors also use a single feature map, while SSD uses multiple feature maps at different scales.

  3. “Faster R-CNN replaced R-CNN and Fast R-CNN” — Faster R-CNN is an evolution, not a replacement. It builds directly on Fast R-CNN’s RoI pooling and detection head. The key contribution is replacing the external Selective Search with the learned RPN. Fast R-CNN’s architecture is essentially preserved in the second stage.

  4. “RoI Pooling is the same as RoI Align” — RoI Pooling (used in Faster R-CNN) quantizes the pooling regions to the feature map grid, causing small misalignments. RoI Align (introduced in Mask R-CNN, 2017) removes this quantization for more precise spatial alignment, which matters for mask prediction.

  5. “The RPN needs a pretrained backbone” — While the paper used pretrained VGG-16/ZF-Net for best results, the RPN itself can be trained from scratch. The paper showed end-to-end training works, though pretraining significantly boosts accuracy on real datasets.

Real Citations

  • Ren, S., He, K., Girshick, R., & Sun, J. (2015). “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.” NeurIPS 2015. Extended version in IEEE TPAMI, 2016. arXiv:1506.01497. 56,000+ citations.
  • Cited by He et al. (2017) “Mask R-CNN” — extends Faster R-CNN with instance segmentation via RoI Align and a mask branch.
  • Cited by Lin et al. (2017) “Feature Pyramid Networks for Object Detection” — adds multi-scale feature pyramid to the Faster R-CNN framework.
  • Cited by Liu et al. (2016) “SSD: Single Shot MultiBox Detector” — adapts the anchor concept to single-stage detection.
  • Cited by Lin et al. (2017) “Focal Loss for Dense Object Detection” (RetinaNet) — addresses the class imbalance problem first identified in the RPN’s anchor matching.
  • Cited by Redmon & Farhadi (2016) “YOLO” — discusses Faster R-CNN’s speed-accuracy tradeoff as motivation for single-stage detection.
  • Won Best Paper Honorable Mention at NeurIPS 2015. Foundation of 1st-place ILSVRC 2015 and COCO 2015 detection tracks.