AI engineering glossary · LLM training alignment
What is PPO (proximal policy optimisation)?
Proximal policy optimisation (also called PPO): A reinforcement learning algorithm that improves a policy in small, stable steps by clipping how far the new policy's probabilities may move from the old ones in each update. It was the standard optimiser in early RLHF pipelines, using a value model to estimate advantages.
Proximal policy optimisation: an example
With clip range 0.2, if a token's probability under the new policy rises to 1.5 times the old, the clipped objective stops rewarding the extra increase beyond 1.2 times.
Related terms
More LLM training alignment terms
- QLoRA
- Rank (LoRA)
- Reasoning model
- Reinforcement learning from AI feedback
- Reinforcement learning from human feedback
- Reinforcement learning with verifiable rewards
- Rejection sampling fine-tuning
- Reward hacking
Where you learn it
Proximal policy optimisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.