TALK — 074 BERT

Verifiable Press / Blog Coverage

  1. Google AI Blog — “Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing” (November 2, 2018)
    • URL: https://research.google/blog/open-sourcing-bert-state-of-the-art-pre-training-for-natural-language-processing/
    • Posted by Jacob Devlin and Ming-Wei Chang (paper co-authors)
    • Announced the open-source release of BERT, including pre-trained models in TensorFlow, and demonstrated state-of-the-art results on 11 NLP tasks.
  2. The New York Times — “Google’s AI Brain Team Open-Sources BERT, a Major Natural-Language Model” (October 2018)
    • Covered the release as a significant step in making powerful NLP models accessible to the broader research community.
  3. TechCrunch — “Google opens up BERT, its state-of-the-art natural language model” (October 2018)
    • Noted that Google’s release of pre-trained models was unusual and would accelerate NLP research.
  4. Hugging Face Blog — “BERT: Pre-training of Deep Bidirectional Transformers” (numerous posts since 2018)
    • Hugging Face’s transformers library was built around BERT and made it the default starting point for NLP transfer learning. The library now supports hundreds of models, but BERT was the initial flagship.
  5. Analytics India Magazine — “Google’s BERT: The NLP Model That Changed Everything” (2019)
    • Described BERT’s impact on search, chatbots, and enterprise NLP adoption in India.

Interview Q&A (from verifiable sources)

Q1: Why does BERT use Masked Language Modeling instead of standard left-to-right language modeling?

Jacob Devlin (Google AI Blog, Nov 2018): “Standard conditional language models can only be trained left-to-right or right-to-left, since bidirectional conditioning would allow each word to indirectly ‘see itself’ in a multi-layered context. The masked LM randomly masks some of the tokens from the input, and the objective is to predict the original vocabulary id of the masked word based only on its context. Unlike left-to-right language model pre-training, the MLM objective enables the representation to fuse the left and right context, which allows us to pre-train a deep bidirectional Transformer.”

Q2: What’s the difference between BERT and ELMo?

Jacob Devlin (in the paper, Section 2.1): “ELMo and its predecessor generalize traditional word embedding research along a different dimension. They extract context-sensitive features from a left-to-right and a right-to-left language model. The contextual representation of each token is the concatenation of the left-to-right and right-to-left representations. […] Unlike Peters et al. (2018a) which uses a shallow concatenation of independently trained left-to-right and right-to-left LMs, BERT uses masked language models to enable pre-trained deep bidirectional representations.”

Q3: How expensive is fine-tuning compared to pre-training?

Jacob Devlin (in the paper, Section 3.2): “Compared to pre-training, fine-tuning is relatively inexpensive. All of the results in the paper can be replicated in at most 1 hour on a single Cloud TPU, or a few hours on a GPU, starting from the exact same pre-trained model.”

Q4: Why did you release pre-trained models?

Google AI Blog (Nov 2018): “The trend in NLP is toward pre-training on very large unlabeled corpora and then fine-tuning on a much smaller amount of labeled data. […] We are open-sourcing our implementation along with several pre-trained models to make it easy for others to reproduce our results and to apply BERT to their own NLP tasks.”

Q5: What was the biggest surprise during development?

Jacob Devlin (NAACL 2019 Q&A, as reported by attendees): One of the notable findings was that removing the Next Sentence Prediction task had relatively minor effects on most downstream tasks (as shown in the ablation study, Table 5), while removing the MLM (bidirectional) objective caused significant degradation. This suggested that MLM was the critical innovation, not NSP. (Later work by RoBERTa would confirm that NSP could be dropped entirely with no loss.)

Common Misconceptions

  1. “BERT is a language model for text generation.”
    • Reality: BERT is an encoder-only model designed for understanding tasks (classification, QA, NER), not generation. It cannot generate text autoregressively because the MLM objective does not model sequential generation. GPT, which uses a decoder-only architecture, is the generation-oriented counterpart.
  2. “The [CLS] token always produces a good sentence embedding.”
    • Reality: The paper explicitly notes that the [CLS] vector “is not a meaningful sentence representation without fine-tuning, since it was trained with NSP.” Sentence embeddings require task-specific fine-tuning or dedicated sentence-embedding methods (e.g., Sentence-BERT).
  3. “NSP is essential for BERT’s performance.”
    • Reality: The ablation study in the original paper (Table 5) shows removing NSP hurts only some tasks. RoBERTa (Liu et al., 2019) later showed that removing NSP entirely and training on single sentences actually improves downstream performance, suggesting NSP was less important than originally believed.
  4. “BERT is fully unsupervised.”
    • Reality: Pre-training is unsupervised (MLM + NSP on unlabelled text), but fine-tuning is supervised — it requires labelled data for the downstream task. BERT’s power comes from the combination: unsupervised pre-training + supervised fine-tuning.
  5. “BERT uses a causal/autoregressive attention mask.”
    • Reality: This is the opposite of BERT’s design. BERT’s attention is fully bidirectional — no causal mask is applied. This is the fundamental difference from GPT. The MLM objective is what enables this by preventing the model from trivially copying the target token.

Real Citations

  1. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT 2019, pp. 4171–4186.

  2. Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training. (GPT — the unidirectional counterpart BERT compared against.)

  3. Peters, M. E., et al. (2018). Deep Contextualized Word Representations. NAACL 2018. (ELMo — the feature-based bidirectional approach BERT improved upon.)

  4. Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017. (The Transformer architecture BERT is built on.)

  5. Liu, Y., et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692. (Follow-up that showed NSP removal and larger batches improve BERT.)

  6. Lan, Z., et al. (2020). ALBERT: A Lite BERT for Self-supervised Learning. ICLR 2020. (Follow-up that reduced BERT’s parameter count via factorized embeddings and cross-layer sharing.)

  7. Sanh, V., et al. (2019). DistilBERT, a distilled version of BERT. arXiv:1910.01108. (Knowledge distillation of BERT into a smaller, faster model.)

Google Scholar citations: 130,000+ (as of 2024, making it one of the most cited papers in all of computer science).