AI engineering glossary · LLM training alignment
What is TRPO (trust region policy optimisation)?
Trust region policy optimisation (also called TRPO): A reinforcement learning algorithm that improves a policy while constraining the KL divergence between old and new policies to stay under a threshold, guaranteeing steadier progress. PPO was designed as a simpler, first-order approximation of the same idea.
Trust region policy optimisation: an example
An update is accepted only if the average KL divergence between the old and new policies is below a small limit such as 0.01.
Related terms
More LLM training alignment terms
- UltraFeedback
- Value function
- Warmup-stable-decay schedule
- Weak-to-strong generalisation
- ZeRO
- 3D parallelism
- 8-bit optimiser
- Adapter
Where you learn it
Trust region policy optimisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.