TALK — WaveNet: A Generative Model for Raw Audio

Press / Blog Coverage

  1. Google DeepMind Blog — “WaveNet: A generative model for raw audio” (Sept 2016): Official announcement with audio samples. The blog post demonstrated that human listeners rated WaveNet-generated English speech as significantly more natural than prior parametric and concatenative TTS systems. URL: https://deepmind.google/discover/blog/wavenet-a-generative-model-for-raw-audio/

  2. The Verge — “Google’s AI can now make its own sounds based on what it sees” (Sept 2016): Covered WaveNet’s audio generation quality, noting the system’s ability to produce speech that sounded remarkably human.

  3. TechCrunch — “DeepMind’s WaveNet uses neural networks to generate human-like speech” (Sept 2016): Discussed the technology’s implications for virtual assistants and TTS applications.

  4. Wired UK — “Google’s DeepMind can now synthesize speech that sounds more human than human” (Sept 2016): Highlighted the subjective quality improvements over existing systems.

  5. Google AI Blog — “A neural network that can generate natural sounding speech” (2017): Follow-up coverage when WaveNet was deployed in Google Assistant, reducing inference time from minutes to real-time.

Interview-Style Q&A

Q: Why use dilated convolutions instead of RNNs for audio generation?

A: RNNs process sequences sequentially, which makes both training and generation slow for very long audio sequences (16,000+ samples per second). Dilated causal convolutions allow parallel training (all timesteps computed simultaneously via teacher forcing) while still maintaining strict causality. The exponentially growing dilation factors give the network a large receptive field — covering hundreds or thousands of samples — with relatively few layers and parameters.

Q: Why mu-law quantization instead of linear quantization or continuous distributions?

A: Raw 16-bit audio has 65,536 possible values per sample, making a softmax output layer prohibitively large. Mu-law companding is a non-linear transformation that allocates more quantization levels to the small-amplitude signals that dominate speech, and fewer to large amplitudes. Quantizing to 256 categories after companding preserves perceptual quality far better than linear 8-bit quantization. A categorical (softmax) distribution is also more flexible than a Gaussian mixture model — it makes no assumptions about the shape of the distribution.

Q: What makes gated activation units (tanh * sigmoid) better than ReLU for audio?

A: The gated activation unit z = tanh(W_f * x) ⊙ sigmoid(W_g * x) combines a filter pathway (tanh) with a learned gating mechanism (sigmoid). This allows the network to selectively pass or suppress information at each layer and timestep, which is important for audio where silence, onset transients, and sustained tones all require different processing. The authors found this worked significantly better than ReLU in initial experiments, likely because audio signals have complex temporal structure that benefits from multiplicative gating.

Q: How slow is generation, and how was this solved?

A: Generation is inherently sequential — each sample must be predicted before it can be fed back to predict the next. With the original WaveNet, generating one second of audio took minutes on a GPU. This was addressed in follow-up work (Parallel WaveNet, 2017) using probability density distillation, where an autoregressive teacher network trains a fully parallel student network. This reduced generation to real-time and eventually enabled deployment in Google Assistant.

Q: Can a single WaveNet model multiple speakers?

A: Yes. By conditioning the model on a speaker identity (one-hot embedding), a single WaveNet can learn the characteristics of many speakers simultaneously and switch between them at generation time. The paper demonstrated this on a multi-speaker dataset, showing the model could generate different voices with equal fidelity.

Common Misconceptions

  1. “WaveNet is an RNN” — No. WaveNet uses only convolutional layers. The autoregressive property comes from causal convolutions, not recurrent connections. This is what enables parallel training.

  2. “Dilated convolutions are the same as strided convolutions” — No. Strided convolutions reduce the output size (like pooling). Dilated convolutions keep the same output size but skip input samples at regular intervals, effectively widening the filter’s view without increasing parameter count.

  3. “WaveNet generates audio in real time” — The original WaveNet was very slow at inference (minutes per second of audio). Real-time generation came later with Parallel WaveNet and subsequent distillation-based approaches.

  4. “You need real speech data to train WaveNet” — While the paper used real speech, the architecture works on any audio signal. Our notebook demonstrates this with synthetic sine wave data.

Real Citations

  • van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016). WaveNet: A Generative Model for Raw Audio. arXiv:1609.03499.
  • van den Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016). Pixel Recurrent Neural Networks. ICML 2016. (PixelCNN — the architectural predecessor that inspired WaveNet)
  • van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016). Conditional Image Generation with PixelCNN Decoders. NeurIPS 2016. (Gated PixelCNN — source of the gated activation unit)
  • He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv:1512.03385. (Residual connections used in WaveNet blocks)
  • van den Oord, A., Li, Y., Babuschkin, I., et al. (2018). Parallel WaveNet: Fast High-Fidelity Speech Synthesis. ICML 2018. (Follow-up enabling real-time generation)
  • Prenger, R., Catanzaro, B. (2019). WaveGlow: A Flow-based Generative Network for Speech Synthesis. arXiv:1811.00002. (Alternative approach to neural vocoding)