AI engineering glossary · LLM training alignment
What is clipped surrogate objective?
Clipped surrogate objective: The PPO loss that multiplies the advantage by the probability ratio between new and old policies, then clips that ratio to the range 1 - epsilon to 1 + epsilon and takes the smaller value. This removes any incentive for a single update to change the policy too much.
Clipped surrogate objective: an example
With epsilon = 0.2, a ratio of 1.4 on a positive-advantage token is treated as 1.2, so the gradient from that token vanishes.
Related terms
More LLM training alignment terms
- Common Crawl
- Compute-optimal training
- Constitutional AI
- Continued pretraining
- Critical batch size
- Data curation
- Data mixture
- Debate (AI safety)
Where you learn it
Clipped surrogate objective is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.