TALK — Progressive Growing of GANs

Press / Blog Coverage

  1. Machine Learning Mastery — “How to Implement Progressive Growing GAN Models in Keras”
    • Step-by-step tutorial implementing ProGAN with fade-in, toRGB/fromRGB layers, and composite models for training.
    • URL: https://machinelearningmastery.com/how-to-implement-progressive-growing-gan-models-in-keras/
  2. Papers With Code — Paper page with method summary and results comparison
    • Lists ProGAN under “Image Generation” methods, tracks benchmark results.
    • URL: https://paperswithcode.com/paper/progressive-growing-of-gans-for-improved
  3. NVIDIA Research — Official NVIDIA research publication page
    • Originally published as an NVIDIA research paper; the official TensorFlow implementation was open-sourced.
    • URL: https://research.nvidia.com/publication/2017-10_Progressive-Growing-of
  4. Google AI Blog — Referenced in discussions of high-quality image generation
    • The paper was widely discussed in the ML community for achieving photorealistic 1024×1024 face generation.
  5. Reddit r/MachineLearning — Significant community discussion upon release
    • The generated celebrity faces were frequently shared and discussed, with many commenters noting the “uncanny valley” quality of the faces.

Citations (verifiable)

  • OpenAlex citation count: ~1,545 (as of 2026)
  • arXiv ID: 1710.10196
  • Published at ICLR 2018
  • Key citing works include:
    • StyleGAN (Karras et al., 2019) — arXiv:1812.04948
    • BigGAN (Brock et al., 2019) — arXiv:1809.11096
    • StyleGAN2 (Karras et al., 2020) — arXiv:1912.04458

Interview Q&A

Q: What was the main motivation for progressive growing? A: The main motivation was training instability. High-resolution GANs were notoriously difficult to train — they would often collapse or produce artifacts. By starting at low resolution and progressively growing, both networks first learn the large-scale structure (which is easier) before tackling fine detail. This naturally curriculum-learns the problem from easy to hard.

Q: Why is the fade-in mechanism important? A: Without fade-in, adding new layers to a trained network causes a sudden shock — the new layers are randomly initialized and can disrupt the learned representations. The fade-in smoothly transitions from the old output to the new output over many training iterations, giving the network time to adapt to the new capacity without forgetting what it already learned.

Q: How does equalized learning rate differ from standard He initialization? A: Standard He initialization sets the weights once at initialization. Equalized learning rate instead initializes weights from a standard normal and dynamically scales them by a constant at every forward pass. This ensures the dynamic range of the weights stays consistent throughout training, preventing some layers from learning much faster than others purely due to their scale.

Q: What is minibatch standard deviation and why does it help? A: It’s a small layer added to the discriminator that computes the standard deviation across all samples in the minibatch for each feature. If the generator is mode-collapsing (producing very similar outputs), the minibatch stddev will be near zero, and the discriminator can easily detect this. This pushes the generator toward producing more diverse outputs.

Q: How does ProGAN compare to StyleGAN? A: StyleGAN (2019) is the direct successor. It keeps the progressive growing training methodology but replaces the sequential latent input with an AdaIN-based style injection mechanism at each layer. This gives much finer control over the generated images (coarse styles at low resolution, fine styles at high resolution) and generally produces higher-quality results.

Common Misconceptions

  1. “Progressive Growing requires transposed convolutions.” — False. The original NVIDIA implementation used transposed convolutions (fractionally-strided convolutions), but nearest-neighbor upsampling followed by a regular convolution is equally valid and often more stable. Many modern implementations use upsampling + conv.

  2. “ProGAN uses Wasserstein loss.” — False. ProGAN uses the standard non-saturating GAN loss. The WGAN and WGAN-GP papers are separate works. Some ProGAN reimplementations experiment with WGAN-GP loss, but the original paper uses the standard GAN objective.

  3. “Progressive Growing is obsolete.” — Not exactly. While diffusion models have largely surpassed GANs for image generation, the progressive growing principle (curriculum from simple to complex) remains influential. StyleGAN2-ADA and StyleGAN3 still use variants of the progressive training approach, and the technique is used in other domains like video generation.

  4. “The fade-in is just a learning rate schedule.” — No, it’s an architectural blend. The alpha parameter blends between the old resolution output (upsampled) and the new resolution output at the pixel level, not at the parameter level.

  5. “ProGAN can only generate faces.” — No. The paper demonstrates results on CelebA, CelebA-HQ, CIFAR-10, and LSUN. The progressive growing technique is domain-agnostic, though it is most famous for the face generation results.