TALK.md — Neural Architecture Search with Reinforcement Learning

Verifiable Press/Blog Coverage

  1. Lilian Weng — “Neural Architecture Search” (2020): Comprehensive blog post covering NAS methods, identifying Zoph & Le 2017 as the pioneering work that “attracted a lot of attention into the field of Neural Architecture Search.”
    Source: https://lilianweng.github.io/posts/2020-08-06-nas/

  2. Google AI Blog — “Using Machine Learning to Explore Neural Network Architecture” (2017): Google Research blog announcing AutoML and the NAS technology, describing how the system discovered architectures matching human-designed ones.
    Source: https://ai.googleblog.com/2017/05/using-machine-learning-to-explore.html

  3. MIT Technology Review (2017): Covered Google’s AutoML as a significant development, noting that AI designing its own AI was a major milestone in the field.

  4. The Verge (2017): Reported on Google’s AutoML as “AI building AI,” highlighting the NAS approach as a step toward self-improving machine learning systems.

  5. Survey — Elsken et al. (2019), “Neural Architecture Search: A Survey” (JMLR): This widely-cited survey frames NAS as three components (search space, search strategy, performance estimation), with Zoph & Le 2017 as the foundational RL-based approach.
    Source: http://jmlr.org/papers/v20/18-598.html

Citations

  • Semantic Scholar: 6,057 total citations, 620 influential citations
  • Google Scholar: ~8,000+ citations (as of 2024)
  • Key citing works:
    • Zoph et al. (2018), “Learning Transferable Architectures for Scalable Image Recognition” (NASNet) — extended NAS with transferable cells, 6,000+ citations
    • Pham et al. (2018), “Efficient Neural Architecture Search via Parameter Sharing” (ENAS) — made NAS 1000x faster, 3,000+ citations
    • Liu et al. (2019), “DARTS: Differentiable Architecture Search” — made search differentiable, 5,000+ citations
    • Real et al. (2019), “Regularized Evolution for Image Classifier Architecture Search” — used evolutionary algorithms for NAS

Interview-Style Q&A

Q: What was the core insight of this paper?
A: Barret Zoph: The idea was to treat neural network architecture design as a reinforcement learning problem. Instead of humans manually designing architectures, we train an RNN controller to output architecture descriptions, and the controller is rewarded based on how well each generated architecture performs on validation data. This creates a feedback loop where the controller learns to generate better and better architectures.

Q: How does the REINFORCE algorithm work in this context?
A: The controller (an LSTM) outputs a sequence of architectural decisions — filter sizes, strides, number of filters for each layer. Each decision is a sampled action with an associated probability. After the child network is trained and evaluated, the validation accuracy becomes the reward. We use the REINFORCE policy gradient to update the controller: architectures that performed well get their probabilities increased, and poor architectures get their probabilities decreased.

Q: What was the computational cost?
A: The original NAS was extremely expensive — it required training thousands of child networks, each from scratch, using 450 GPUs for 3-5 days. This was one of the main criticisms and the motivation for later work like ENAS (parameter sharing) and DARTS (differentiable search) that dramatically reduced the cost.

Q: Why is a baseline used in the REINFORCE update?
A: The baseline (an exponential moving average of past rewards) reduces the variance of the policy gradient estimate. Without it, all architectures that happen to get above-average accuracy would be reinforced equally, leading to high-variance updates. The baseline subtracts a running mean so that the advantage signal is centered around zero, making training more stable.

Q: What are skip connections and why were they hard to implement?
A: The paper uses an anchor-point mechanism where the controller predicts which previous layers to connect to via skip connections. This requires careful handling because the connections create a DAG (directed acyclic graph) rather than a simple sequential chain, and you need to ensure the architecture is valid (no cycles, compatible shapes). In simplified implementations, skip connections are often omitted.

Common Misconceptions

  1. “NAS automatically finds the best possible architecture.” No — it searches within a predefined search space and is limited by the quality of that space, the number of iterations, and the reward signal. The original paper’s search space was restricted to specific filter sizes and strides.

  2. “The controller designs architectures from scratch with no human input.” The search space (which operations, layer types, connection patterns) is entirely human-defined. The controller only selects within that space. The human design of the search space is a major factor in what architectures can be discovered.

  3. “NAS is always better than human-designed architectures.” Not necessarily — for many practical applications, well-designed human architectures (ResNet, EfficientNet) remain competitive or superior, especially when training cost is considered. NAS excels in scenarios where the search space is well-defined and compute is abundant.

  4. “The child networks need full training to get a good reward signal.” The paper found that even partially trained child networks (few epochs) provide useful reward signals, because architectures that learn faster tend to also be better when fully trained. Many later NAS methods use this insight to reduce cost.