Highway Networks
Paper: Highway Networks by Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber (IDSIA). Presented at the ICML 2015 Deep Learning Workshop. Full version: Training Very Deep Networks (arXiv:1507.06228). arXiv link: https://arxiv.org/abs/1505.00387
Summary. Highway networks were the first practical demonstration that feed-forward neural networks with hundreds of layers could be trained end-to-end with ordinary SGD. The key idea is to let each layer learn a smooth, data-dependent blend between transforming its input and simply carrying it forward. For every block, two functions are computed: a block state H(x) (a normal non-linear transformation) and a transform gate T(x) = σ(W_T x + b_T). The layer then outputs y = H(x)·T(x) + x·(1 − T(x)). Because the transform gate is bounded in [0, 1], the layer can act like a plain layer when T ≈ 1, act like an identity (“highway”) when T ≈ 0, or interpolate anywhere in between. This gating mechanism is borrowed from LSTM memory cells, and the crucial initialization trick is to set the transform-gate bias to a negative value (e.g. −2 or −4) so the network starts by passing information forward almost unchanged. The authors showed that 100-layer highway MLPs optimized well on MNIST while plain MLPs of the same depth and parameter count stalled, and later trained 900-layer highway networks on CIFAR-100 without difficulty. Highway networks are a direct precursor to ResNet: ResNet’s y = F(x) + x is a special case in which the carry path is always fully open.
What problem does it solve
Imagine you are trying to pass a whispered message around a classroom by having every student whisper it to the next. With only two or three students the message usually arrives intact, but with twenty students the message becomes a garbled mess. That garbling is exactly what happens to gradients and useful signals when you stack dozens of plain neural-network layers: the signal gets distorted or squeezed away, and the network becomes very hard to train.
Highway Networks solve this by giving the classroom an intercom system. Now each student can choose either to repeat the message unchanged (carry it forward on the highway) or to add their own small summary (transform it). At first every student is told to just repeat the message, so nothing gets lost. As training progresses, some students learn when it is safe to add useful details, and others keep passing the original message through. The result is that you can have many, many “students” (layers) in the chain without the message dying, and the network can be trained with ordinary methods.
Resources
Kaggle Notebook:
| File | Description |
|---|---|
| README.md | This overview — a 5-minute read covering the paper’s summary, core idea, key method details, influence, and a plain-English explanation of the problem it solves |
| solution.ipynb | Complete from-scratch PyTorch implementation of the paper’s architecture, with training loops, visualizations, and worked examples |
| CODE_ARCHITECTURE.md | Section-by-section breakdown of the notebook — what each cell does, data flow, tensor shapes, and key design decisions |
| TALK.md | Press coverage, blog posts, interview Q&A, citations, and common misconceptions about the paper |