PPO — Press Coverage, Interviews, Misconceptions, Citations

Verifiable Press & Blog Coverage

  1. OpenAI Blog — “Proximal Policy Optimization Algorithms” (August 2017) OpenAI published a blog post introducing PPO alongside the paper, describing it as their default reinforcement learning algorithm. The post includes benchmark results and notes that PPO strikes a balance between performance and ease of implementation.
    • https://openai.com/research/openai-baselines-ppo
  2. OpenAI Blog — “OpenAI Five” (2018) OpenAI Five, the Dota 2 AI system, used PPO as its core RL algorithm. The blog describes training on tens of thousands of CPUs with PPO over months.
    • https://openai.com/research/openai-five
  3. OpenAI Blog — “Learning to Summarize from Human Feedback” / InstructGPT (2022) The InstructGPT paper (Ouyang et al., 2022) — the direct precursor to ChatGPT — uses PPO for the RLHF fine-tuning stage. This is one of the most consequential applications of PPO.
    • https://openai.com/research/instruction-following
  4. Hugging Face Blog — “Illustrating Reinforcement Learning” / CleanRL documentation CleanRL and Hugging Face’s Deep RL Course both use PPO as the primary teaching algorithm, noting its status as the most widely-used on-policy RL method.
    • https://huggingface.co/blog/deep-rl-ppo
    • https://cleanrl.solmer.dev/rl-algorithms/ppo/
  5. Lilian Weng’s Blog — “Policy Gradient Algorithms” (2018) Lilian Weng (OpenAI) wrote a widely-cited survey that covers PPO in detail, explaining the clipped surrogate objective with clear diagrams.
    • https://lilianweng.github.io/posts/2018-04-08-policy-gradient/

Interview Q&A

Q: Why did you choose clipping over the KL penalty approach? A: In the paper’s experiments (Table 1), the clipped surrogate objective with ε=0.2 achieved the highest average normalized score (0.82) across 21 runs on 7 environments, outperforming both adaptive KL penalty (best: 0.74) and fixed KL penalty (best: 0.72). The authors note: “we’ve included [KL penalty] here because it’s an important baseline” but found “the KL penalty performed worse than the clipped surrogate objective.”

Q: How does PPO compare to TRPO in terms of implementation complexity? A: The paper states PPO “has some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically).” PPO requires only first-order optimization (Adam), while TRPO needs conjugate gradient with a Fisher-vector product and line search. The paper also notes TRPO “is not compatible with architectures that include noise (such as dropout) or parameter sharing.”

Q: Can you do multiple gradient steps on the same data with PPO? A: Yes — this is a key advantage. The paper explicitly contrasts with “standard policy gradient methods [which] perform one gradient update per data sample” and proposes “a novel objective function that enables multiple epochs of minibatch updates.” The clip ensures these repeated updates don’t push the policy too far.

Q: What is the role of the entropy bonus? A: The entropy term Sπ_θ is added to the objective “to ensure sufficient exploration, as suggested in past work.” With coefficient c2=0.01 (Table 5 for Atari), it encourages the policy to maintain stochasticity rather than collapsing to a deterministic policy prematurely.

Q: Why not just use a fixed KL penalty? A: The paper explains: “it is hard to choose a single value of β that performs well across different problems — or even within a single problem, where the characteristics change over the course of learning.” The adaptive KL penalty adjusts β based on observed KL divergence, but clipping was still found to perform better empirically.

Common Misconceptions

  1. “PPO is on-policy, so you can’t reuse old data.” Correction: PPO DOES reuse collected data — for K epochs of minibatch updates. The “on-policy” constraint is that data must come from the current (or just-updated) policy. The clip allows safe reuse of the same batch multiple times because it limits how far the policy can move from π_θold.

  2. “The clip prevents the policy from changing at all.” Correction: The clip only removes the incentive for changes that push r beyond [1−ε, 1+ε]. The policy can still change; it just doesn’t get gradient reward for over-extending. The min() ensures that if a change makes things worse (even within the clip range), the full penalty applies.

  3. “PPO requires a GPU.” Correction: PPO on simple environments like CartPole trains in seconds on a CPU. The original paper’s MuJoCo experiments used modest MLPs (2×64). GPU helps for Atari (CNN policy) or large-scale parallel environments.

  4. “PPO is just TRPO with clipping instead of KL constraint.” Correction: While the motivation is similar, PPO is fundamentally different. TRPO solves a constrained optimization problem with second-order methods. PPO uses unconstrained first-order optimization with a modified objective. The clip is part of the loss function, not a constraint — so standard Adam SGD works directly.

  5. “The clip ratio ε=0.2 is a hard limit on policy change.” Correction: ε controls the clip on the probability ratio r, not directly on parameter change. The actual KL divergence can exceed what you’d naively expect from ε because the ratio is computed per-sample and the policy is a distribution. The clip is a heuristic that works well empirically.

Real Citations

  • Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
  • Schulman, J., Levine, S., Moritz, P., Jordan, M., & Abbeel, P. (2015). Trust Region Policy Optimization. ICML. (PPO’s predecessor)
  • Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv:1506.02438. (GAE, used by PPO)
  • Mnih, V., et al. (2016). Asynchronous Methods for Deep Reinforcement Learning. ICML. (A3C/A2C, compared against)
  • Ouyang, L., et al. (2022). Training Language Models to Follow Instructions with Human Feedback. arXiv:2203.02155. (InstructGPT — uses PPO for RLHF)
  • Brockman, G., et al. (2016). OpenAI Gym. arXiv:1606.01540. (Environment framework)
  • Kakade, S. & Langford, J. (2002). Approximately Optimal Approximate Reinforcement Learning. ICML. (CPI — the original surrogate objective)