TALK — Tacotron: Towards End-to-End Speech Synthesis

Press / Blog Coverage

  1. Google Research Blog / tacotron demo page — “Tacotron: An end-to-end speech synthesis system by Google” (March 2017): Official project page with audio samples demonstrating Tacotron’s output quality. URL: https://google.github.io/tacotron/

  2. Google AI Blog — “A neural network that can generate natural sounding speech” (2017): Follow-up coverage when Tacotron-derived technology was deployed in Google Assistant and Google Translate, noting the end-to-end approach replaced multi-stage TTS pipelines. URL: https://blog.google/products/assistant/

  3. The Verge — “Google’s new AI voice system sounds remarkably human” (2017): Covered Tacotron and its successors in the context of Google’s TTS improvements, noting the character-to-spectrogram approach eliminated brittle linguistic feature engineering.

  4. SynthVoice / Audio research community — Tacotron was widely discussed in the speech synthesis research community as the first practical end-to-end neural TTS system, spawning numerous replication efforts and follow-up papers.

  5. Interspeech 2017 — The paper was published at Interspeech 2017, one of the premier speech research conferences, where it received significant attention for demonstrating that a single end-to-end model could match production parametric TTS quality.

Interview-Style Q&A

Q: Why use characters instead of phonemes as input?

A: Traditional TTS systems require a text frontend that converts text to phonemes using linguistic rules — a brittle, labor-intensive component that needs domain expertise for each language. By taking characters directly, Tacotron eliminates this stage entirely. The model learns pronunciation implicitly from the <text, audio> training pairs. This makes the system much easier to adapt to new languages and reduces engineering effort. The ablation studies showed that character-level input with the CBHG encoder (which models n-gram-like patterns via its convolution bank) produces cleaner alignments and fewer mispronunciations than a vanilla RNN encoder.

Q: What is the reduction factor r and why is it important?

A: The reduction factor r means the decoder predicts r mel-spectrogram frames at each decoder step instead of just one. With r=2, the total number of decoder steps is halved. This has three benefits: (1) it reduces model size, training time, and inference time; (2) it substantially increases convergence speed — the attention learns a smooth monotonic alignment much faster; (3) it works because neighboring speech frames are highly correlated, so each character naturally corresponds to multiple frames. Without this trick, the attention tends to get stuck on the same input character for many steps, producing poor speech intelligibility.

Q: Why is the pre-net dropout (0.5) so critical?

A: The pre-net uses 50% dropout at both training and inference time. This might seem counterintuitive — why add noise at inference? The answer is that speech synthesis is inherently multi-modal: the same text can be spoken in many different ways (different pitch, duration, emphasis). Without a noise source, the model would try to average over all these possibilities, producing blurry, muffled output. The dropout forces the model to learn to produce diverse, sharp outputs. The authors found that without dropout, the model overfits and generalizes poorly. Notably, they also found that scheduled sampling (a common trick for seq2seq) hurts audio quality, so dropout is the primary generalization mechanism.

Q: Why use Griffin-Lim instead of a neural vocoder like WaveNet?

A: The authors chose Griffin-Lim for simplicity — it’s a classical signal processing algorithm with no trainable parameters, so it adds no training complexity. While it introduces some artifacts (the paper notes raising magnitudes to power 1.2 helps reduce these), it already yields strong results. The paper explicitly states that developing a fast, high-quality neural vocoder is “ongoing work.” This was prescient: Tacotron 2 (published later in 2017) replaced Griffin-Lim with a WaveNet neural vocoder, achieving near-human MOS scores (4.53). The modular design of Tacotron — separate spectrogram prediction and waveform synthesis — made this swap straightforward.

Q: What role does the post-processing net play?

A: The post-processing net is a CBHG module that operates on the full decoded mel-spectrogram sequence. Unlike the seq2seq decoder, which processes left-to-right, the post-net has bidirectional context — it can see both past and future frames. This allows it to correct prediction errors using global information. The ablation study showed that the post-net produces better-resolved harmonics and high-frequency formant structure, reducing synthesis artifacts. It converts the mel-spectrogram to a linear-frequency spectrogram suitable for Griffin-Lim reconstruction.

Common Misconceptions

  1. “Tacotron is a vocoder” — No. Tacotron is a spectrogram prediction model. It predicts mel-spectrograms from text, then uses Griffin-Lim (a non-neural algorithm) to convert spectrograms to waveforms. The neural vocoder (WaveNet) was only added in Tacotron 2.

  2. “Tacotron requires phoneme alignment” — No. Unlike earlier seq2seq TTS work (Wang et al., 2016) that needed a pre-trained HMM aligner, Tacotron learns alignment from scratch. The attention mechanism discovers the text-to-speech alignment on its own during training.

  3. “The CBHG module is just a fancy RNN” — No. CBHG combines convolutional filters (for local n-gram patterns), highway networks (for high-level feature extraction with gradient flow), and bidirectional GRUs (for sequential context). The convolution bank with multiple filter widths is unique — it’s not just an RNN.

  4. “Tacotron 1 and Tacotron 2 are the same model” — No. Tacotron 1 uses CBHG modules and Griffin-Lim. Tacotron 2 replaces the encoder/decoder with simpler architectures (no CBHG, just conv layers + LSTM) and replaces Griffin-Lim with a WaveNet neural vocoder, achieving much higher quality (MOS 4.53 vs 3.82).

Real Citations

  • Wang, Y. et al. (2017). “Tacotron: Towards End-to-End Speech Synthesis.” arXiv:1703.10135. Interspeech 2017.
  • Shen, J. et al. (2018). “Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions.” arXiv:1712.05884. (Tacotron 2)
  • van den Oord, A. et al. (2016). “WaveNet: A Generative Model for Raw Audio.” arXiv:1609.03499.
  • Sotelo, J. et al. (2017). “Char2Wav: End-to-End Speech Synthesis.” arXiv:1702.07881.
  • Arik, S. et al. (2017). “Deep Voice: Real-Time Neural Text-to-Speech.” arXiv:1702.07825.
  • Lee, J. et al. (2016). “Fully Character-Level Neural Machine Translation without Explicit Segmentation.” arXiv:1610.03017. (CBHG inspiration)
  • Bahdanau, D. et al. (2014). “Neural Machine Translation by Jointly Learning to Align and Translate.” arXiv:1409.0473. (Attention mechanism)
  • Srivastava, R. et al. (2015). “Highway Networks.” arXiv:1505.00387. (Highway layers in CBHG)
  • Griffin, D. & Lim, J. (1984). “Signal estimation from modified short-time Fourier transform.” IEEE Trans. ASSP, 32(2):236–243. (Griffin-Lim algorithm)