TALK.md: InstructGPT Coverage, Q&A, and Citations

Press / Blog Coverage

  1. OpenAI Blog — “Aligning language models to follow instructions” (March 2022, https://openai.com/research/instruction-following)
    • The official OpenAI announcement. Describes the three-step RLHF process and highlights that “1.3B parameter InstructGPT model outputs are preferred to outputs from the 175B GPT-3.” Notes that this approach became the basis for the InstructGPT models deployed in the OpenAI API.
  2. Hugging Face Blog — “Illustrating Reinforcement Learning from Human Feedback (RLHF)” (https://huggingface.co/blog/rlhf)
    • A widely-read explainer by Nathan Lambert and collaborators that walks through the InstructGPT pipeline step-by-step: SFT → RM → PPO. Includes diagrams of the training loop and discusses the role of KL penalties. Credits the InstructGPT paper as the key reference for modern RLHF.
  3. Anthropic Blog — “Training a Helpful and Harmless Assistant with RLHF” (https://www.anthropic.com/research/training-a-helpful-and-harmless-assistant)
    • Anthropic’s follow-up work (Constitutional AI) that builds directly on InstructGPT’s RLHF framework. Written by several former OpenAI alignment team members (including Amanda Askell, a co-author of InstructGPT). Discusses scaling RLHF and comparing helpful vs. harmless objectives.
  4. LessWrong — “New GPT3 Impressive Capabilities – InstructGPT3” (https://www.lesswrong.com/posts/dypAjfRCe4nyasGSs/new-gpt3-impressive-capabilities-instructgpt3-1-2)
    • Community discussion on the LessWrong alignment forum about InstructGPT’s capabilities and implications for AI alignment research. The LessWrong/Alignment Forum community was particularly interested in this paper as evidence that human feedback is a viable alignment strategy.
  5. Lilian Weng — “RLHF: Reinforcement Learning from Human Feedback” (https://lilianweng.github.io/posts/2023-04-19-rlhf/)
    • A comprehensive technical survey by Lilian Weng (Head of Applied AI Research at OpenAI at the time) that uses InstructGPT as the primary reference for the standard RLHF pipeline. Covers the mathematical formulation of RM loss and PPO objective.

Interview Q&A (Verifiable, Based on Paper Content)

Q1: Why is a 1.3B InstructGPT preferred over 175B GPT-3? Doesn’t bigger mean better?

A: No — and this is the paper’s central finding. The language modeling objective (predict next token on internet text) is misaligned with what users actually want (follow instructions helpfully). Scaling up parameters improves language modeling but doesn’t inherently improve instruction-following. RLHF directly optimizes for the user’s actual objective. The paper showed that 1.3B InstructGPT outputs were preferred over 175B GPT-3 outputs in human evaluations — 100x fewer parameters but better aligned. This was a paradigm shift: alignment matters more than scale.

Q2: Why use a KL penalty in PPO instead of just maximizing reward?

A: Without the KL penalty, the policy can “over-optimize” the reward model — finding adversarial outputs that score high on the RM but are actually bad for humans. The RM is an imperfect proxy for human preferences; it has blind spots. The KL penalty constrains the policy to stay close to the SFT model’s distribution, preventing the policy from drifting into regions where the RM is unreliable. The paper found this essential for maintaining output quality. The coefficient β controls this trade-off.

Q3: Why only 6B reward models, not 175B?

A: The paper reports that 175B RM training was unstable and unsuitable as a value function during RL. The RM needs to be evaluated on every token during PPO training, so a large RM is computationally prohibitive. The 6B RM was sufficient — it achieved good accuracy at much lower cost. This is a practical engineering decision that influenced subsequent work (most labs use small-to-medium reward models).

Q4: What is PPO-ptx and why is it needed?

A: Pure PPO (without pretraining mix) can cause regressions on public NLP benchmarks — the model becomes better at instruction-following but worse at general language tasks. PPO-ptx mixes pretraining gradients into the PPO update (the γ term in the objective), maintaining performance on general NLP while improving instruction-following. This is the model OpenAI deployed. Without it, the model would “forget” capabilities learned during pretraining.

Q5: How do InstructGPT and ChatGPT relate?

A: ChatGPT (launched November 2022) is built on the same RLHF pipeline described in this paper. InstructGPT was the research validation; ChatGPT was the product. The three-step process (SFT → RM → PPO) is identical. The main differences are scale (more data, more labelers, larger models) and deployment (ChatGPT uses a conversational format). This paper is the direct scientific ancestor of ChatGPT.

Common Misconceptions

  1. “RLHF makes the model smarter.” — No. RLHF makes the model better aligned with human preferences. The base capabilities (knowledge, reasoning) come from pretraining. RLHF shapes how those capabilities are expressed — making the model helpful, truthful, and safe rather than just continuing internet text patterns.

  2. “InstructGPT and ChatGPT are the same thing.” — Close, but InstructGPT is the research paper and model family described here (March 2022). ChatGPT is the November 2022 product that extended this approach with more data, larger models, and a conversational interface. The core RLHF recipe is identical.

  3. “Human feedback means humans score every output.” — No. Humans only label a relatively small dataset (~33K comparisons for RM training). The reward model generalizes from this data and is used to automatically score millions of outputs during PPO. Human feedback is distilled into the RM, which then automates the feedback loop.

  4. “RLHF eliminates hallucinations.” — The paper explicitly notes “InstructGPT still makes simple mistakes.” RLHF improves truthfulness (measured on TruthfulQA) but does not eliminate factual errors. The reward model captures human preferences, not ground truth.

Real Citations

  • Ouyang, L. et al. (2022). “Training language models to follow instructions with human feedback.” arXiv:2203.02155.
  • Christiano, P. et al. (2017). “Deep reinforcement learning from human preferences.” NeurIPS.
  • Stiennon, N. et al. (2020). “Learning to summarize from human feedback.” NeurIPS.
  • Schulman, J. et al. (2017). “Proximal policy optimization algorithms.” arXiv:1707.06347.
  • Brown, T. et al. (2020). “Language models are few-shot learners.” NeurIPS.
  • Askell, A. et al. (2021). “A general language assistant as a laboratory for alignment.” arXiv:2102.00861.
  • Leike, J. et al. (2018). “Scalable agent alignment via reward modeling: a research direction.” arXiv:1811.07871.