AI engineering glossary · LLM training alignment
What is advantage function?
Advantage function: A measure of how much better an action is than the average action in that situation, computed as expected return of the action minus the value of the state. Subtracting this baseline lowers the variance of policy-gradient updates, so learning is steadier.
Advantage function: an example
If an answer earns reward 0.9 and the model would normally earn 0.6 for that prompt, the advantage is +0.3, so the answer is reinforced.
Related terms
More LLM training alignment terms
- Alignment
- Alignment tax
- Alpaca
- Base model
- Bradley-Terry model
- Causal language modelling
- Chat template
- Chinchilla scaling
Where you learn it
Advantage function is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.