AI-generated review 5 min read Intermediate 928 words

PPO: Proximal Policy Optimization

PPO stabilizes policy gradients with a clipped objective, becoming the default RL algorithm behind robotics, games and RLHF fine-tuning.

ppo policy-gradient reinforcement-learning rlhf trust-region
Reinforcement learning · 2017

Primary source: Proximal Policy Optimization Algorithms (arXiv:1707.06347)

Policy gradient methods train agents by climbing the gradient of expected reward, but they have a dangerous failure mode: one oversized update can destroy a policy that took millions of steps to learn. Early deep reinforcement learning coped with tiny step sizes or elaborate constraints, which made training slow and fiddly. The 2017 paper Proximal Policy Optimization Algorithms by Schulman and colleagues at OpenAI offered a better trade-off — an algorithm nearly as stable as trust-region methods yet as simple to implement as vanilla policy gradients. PPO quickly became the default on-policy algorithm, powering results from robot locomotion to game playing and later fine-tuning language models with RLHF.

Why policy gradients need a trust region

A policy gradient agent collects trajectories with its current policy, estimates how good each action was, and nudges the policy toward better actions. The catch is that the data comes from the old policy, so after a large update the gradient estimates no longer describe the new policy — yet the optimizer keeps pushing in the same direction. Repeated oversized steps can move the policy somewhere performance collapses and recovery is slow. DQN faced a related instability on the value side and answered it with target networks and replay buffers; policy methods needed their own stabilizer. The 2015 TRPO algorithm solved this by constraining each update inside a trust region measured by KL divergence, with monotonic improvement guarantees in theory. But TRPO needs conjugate-gradient second-order math that is awkward to implement and hard to combine with recurrent or very large models. PPO keeps the trust-region spirit while replacing the hard constraint with a first-order trick that takes an afternoon to code.

The clipped surrogate objective explained

PPO measures how far the policy moved with the probability ratio r_t(\theta) = \pi_\theta(a_t|s_t) / \pi_{old}(a_t|s_t), which equals 1 when nothing changed. The classic surrogate objective multiplies this ratio by the estimated advantage A_t, but PPO clips the ratio to stay near 1:

L^{CLIP}(\theta) = \mathbb{E}_t[\min(r_t(\theta) A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t)]

with \epsilon usually set to 0.2. When the advantage is positive, clipping caps the gain from pushing the ratio above 1+\epsilon; when it is negative, it caps the gain from pushing below 1-\epsilon. Taking the minimum keeps the pessimistic bound, so the policy moves only while the clipped and unclipped objectives agree. The paper also tests a KL-penalized variant with an adaptive coefficient, but the clipped version matches or beats it and became the standard everyone implements.

Step 1 Old policy collects rollouts Step 2 Estimate advantages Step 3 Optimize clipped loss, K epochs Step 4 New policy replaces old repeat with fresh on-policy data
PPO loop: collect rollouts with the old policy, estimate advantages, optimize the clipped surrogate for several epochs, then repeat with fresh data.

How PPO training works in practice

PPO is on-policy with reuse: it gathers a batch of trajectories from several parallel environments, computes advantages with generalized advantage estimation, then runs several epochs of minibatch SGD on that same batch before discarding it. The full loss combines the clipped policy term with a value-function error and an entropy bonus that keeps exploration alive:

L(\theta) = L^{CLIP}(\theta) - c_1 L^{VF}(\theta) + c_2 S(\theta)

Typical settings collect around 2048 steps per environment, use minibatches of roughly 64, optimize for about 10 epochs, and train with Adam. Common refinements include normalizing advantages within each batch, clipping the value loss the same way as the policy loss, and annealing the learning rate. Because everything is first-order, PPO drops into any autodiff framework and works with feed-forward, convolutional, and recurrent policies alike, which helped it spread fast.

PPO results on Atari and continuous control

The paper benchmarks PPO against A2C, TRPO, and cross-entropy baselines on Atari games and MuJoCo locomotion tasks such as HalfCheetah, Hopper, and Walker. PPO matches or beats TRPO while being far simpler, and it clearly outperforms vanilla policy gradients. On continuous control it learns stable gaits within roughly an hour of simulated experience on a single machine, where earlier methods needed careful per-task tuning. The authors stress robustness: a single set of hyperparameters works across the whole suite, which was rare in 2017-era deep RL. Sample efficiency also improves because each batch of experience supports multiple epochs of updates instead of a single gradient step. That combination of stability and simplicity is why standard baselines libraries and robotics labs adopted PPO almost immediately.

PPO legacy: from game AI to RLHF alignment

PPO's biggest wins arrived after the paper. OpenAI Five used a large-scale PPO variant to defeat world champions at Dota 2, and robotics teams applied it to dexterous manipulation and legged locomotion. Its most influential role, though, is in language models: InstructGPT fine-tunes GPT with PPO against a learned reward model, the recipe behind ChatGPT's helpfulness. Compared with asynchronous actor-critic updates or off-policy Q-learning, PPO's clipped steps tolerate the noisy, shifting rewards of human feedback without collapsing. Researchers continue to study its quirks — value clipping, KL drift, and sensitivity to code-level details — but nearly a decade on, PPO remains the first algorithm practitioners reach for on a new on-policy problem.

Source

  • Schulman et al., Proximal Policy Optimization Algorithms, 2017. arXiv:1707.06347
  • Schulman et al., Trust Region Policy Optimization, 2015. arXiv:1502.05477
  • Mnih et al., Asynchronous Methods for Deep Reinforcement Learning, 2016. arXiv:1602.01783

Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.

Cite this report
@misc{knowyourml-ppo-proximal-policy-optimization
  title        = {PPO: Proximal Policy Optimization},
  author       = {{Know Your ML}},
  year         = {2017},
  howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=ppo-proximal-policy-optimization}},
  note         = {AI-generated review of arXiv:1707.06347. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit — decision matrix, scorecards and trade-off analysis
Toolkit 7 decision tools Model-agnostic

Machine Learning Decision Toolkit

Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.

$24 View on Gumroad

References

Frequently asked questions

Why do policy gradient methods need clipping?

Vanilla policy gradients can take one oversized step that destroys the policy, because updates computed from old-policy data keep pushing past the point where they are valid. PPO prevents this by clipping the probability ratio, so each update stays inside a safe neighborhood of the old policy.

How does PPO differ from TRPO?

TRPO enforces a hard KL-divergence constraint using second-order conjugate-gradient math, which is complex to implement. PPO approximates the same trust-region behavior with a first-order clipped objective that matches TRPO's performance with far simpler code.

What does the epsilon hyperparameter control?

The epsilon hyperparameter, usually 0.2, sets how far the new policy's action probabilities may move from the old policy's in one update. Smaller values are safer but slower; 0.2 is the robust default from the original paper.

Why is PPO used in RLHF and ChatGPT?

InstructGPT and ChatGPT-style models use PPO to fine-tune language models against a learned reward model of human preferences. PPO's stable clipped updates tolerate the noisy, shifting rewards of human feedback without collapsing.

How to Remain Valuable When Intelligence Becomes Cheap — book cover
eBook 224 pages

How to Remain Valuable When Intelligence Becomes Cheap

This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?

Read the eBook