Dropout — Improving Neural Networks by Preventing Co-adaptation of Feature Detectors

Dropout banner Kaggle

Summary

Large neural networks trained on small datasets have a chronic problem: they memorize the training examples instead of learning general rules, so they perform poorly on new test data. This is called overfitting. Hinton et al. proposed a surprisingly simple fix: during training, randomly drop out (set to zero) roughly half of the neurons in the network for every training example. Each update trains a different “thinned” sub-network, which forces the network to learn robust features that work in many different contexts rather than relying on specific co-adapted combinations of neurons.

What Problem Does It Solve

Imagine you’re studying for a big exam by memorizing the exact questions from your practice test instead of understanding the concepts. You ace the practice test, but on the real exam with new questions, you fail. That’s called overfitting — your brain memorized instead of learned.

Neural networks have the exact same problem. When you train a big neural network on a small amount of data, it memorizes the training examples instead of learning the general patterns. So it does great on the data it’s seen, but badly on new data.

Dropout solves this by randomly turning off about half the “brain cells” (neurons) every time the network looks at a new training example. It’s like a soccer team where the coach randomly benches half the players at every practice — no player can rely on a specific teammate always being there, so every player has to learn to be good on their own. By the time the real game comes, the whole team is stronger because each player learned to handle many different situations independently.

Core Idea

The breakthrough insight is that neurons should not depend on each other too much. In a standard network, a hidden neuron can become useful only because another specific neuron is present; this is “co-adaptation.” Dropout breaks these co-adaptations by randomly deleting neurons on every forward pass. Because the network never knows which neurons will be available, each neuron must learn to detect a feature that is independently useful across a wide variety of internal contexts. At test time, all neurons are used but their outgoing weights are scaled down by the dropout probability, which approximates the effect of averaging over all the thinned networks.

Key Method Details

Training-Time Dropout

  • For each training example, each hidden unit is kept active with probability p (typically p = 0.5 for hidden layers, p = 0.8 for input layers).
  • A binary mask r is sampled from a Bernoulli distribution with probability p and multiplied element-wise with the layer’s output: y = r * f(Wx + b).
  • During backpropagation, gradients flow only through the active units.

Test-Time Weight Scaling

  • At inference time, dropout is turned off (all units are active).
  • To approximate averaging over many thinned networks, outgoing weights from a layer with dropout probability p are multiplied by p.
  • For example, if hidden units were retained 50% of the time, their output weights are halved at test time.

Why It Works

  • Ensemble effect: Dropout trains an exponential number of smaller networks and approximately averages their predictions.
  • Robust features: Each unit must be useful on its own, not just in a specific combination with other units.
  • Regularization: It acts as a strong regularizer, reducing the need for other forms of regularization such as weight decay or early stopping.

Training Setup (from the paper)

  • Tested on MNIST, CIFAR-10, ImageNet, and speech recognition tasks.
  • For MNIST: fully connected networks with ReLU/sigmoid activations.
  • For ImageNet: AlexNet-style convolutional networks with dropout in fully connected layers.
  • Optimizer: SGD with momentum, learning rate tuned per task.

Simplifications in This Notebook

  • Fashion-MNIST instead of the paper’s MNIST, TIMIT, Reuters, CIFAR-10 and ImageNet experiments
  • A small MLP (784 → 256 → 128 → 10, ReLU) trained with Adam (learning rate 1e-3) for 25 epochs
  • PyTorch’s nn.Dropout, which scales the kept units up by 1/(1−p) during training instead of halving the weights at test time (the result is the same on average)
  • No max-norm weight constraint (the paper uses one)

What the Notebook Result Shows

After 25 epochs, the model without dropout reached 95.04% train accuracy and 89.17% test accuracy (a gap of 5.87 points). The model with dropout (p = 0.5) reached 90.85% train and 88.22% test accuracy (a gap of 2.63 points).

So dropout cut the overfitting gap by more than half, and its final test loss was lower (0.3235 vs 0.3661). Without dropout, the test loss stopped improving at about 0.32 (epochs 9 to 16) and then rose to 0.3661, while the train loss kept falling: a clear sign of overfitting. But in this short run, dropout did not give higher test accuracy: it was about 1 point lower. The paper’s big gains came from larger networks trained for much longer.

Influence

Dropout became one of the most widely used regularization techniques in deep learning. It is simple to implement, adds almost no computational cost, and dramatically improves generalization across vision, speech, and NLP tasks. The 2012 ImageNet win by AlexNet used dropout in the fully connected layers, helping establish deep learning as the dominant approach in computer vision. The follow-up JMLR paper (2014) provided a more thorough analysis and is one of the most cited deep-learning papers of all time.

  • Batch Normalization (Ioffe & Szegedy, 2015) — reduces internal covariate shift and, in some cases, reduces the need for dropout.
  • DropConnect (Wan et al., 2013) — randomly drops weights instead of activations.
  • Spatial Dropout (Tompson et al., 2015) — drops entire feature maps in convolutional networks.
  • Monte Carlo Dropout (Gal & Ghahramani, 2016) — reinterpreted dropout as approximate Bayesian inference for uncertainty estimation.
  • Stochastic Depth (Huang et al., 2016) — randomly drops residual blocks during training.
  • Dropout remains a standard baseline regularizer, especially for fully connected layers.

Resources

File Description
README.md This overview — a 5-minute read covering the paper’s summary, core idea, key method details, influence, and a plain-English explanation of the problem it solves
solution.ipynb Complete from-scratch PyTorch implementation of the paper’s architecture, with training loops, visualizations, and worked examples
CODE_ARCHITECTURE.md Section-by-section breakdown of the notebook — what each cell does, data flow, tensor shapes, and key design decisions
TALK.md Press coverage, a talk, Q&A (answers from the paper), and common misconceptions about the paper

Table of contents