AI engineering glossary · LLM training alignment
What is policy gradient?
Policy gradient (also called REINFORCE): A family of reinforcement learning methods that directly adjust a policy's parameters to make actions with high reward more likely. The basic estimate multiplies the gradient of the log-probability of an action by the return it earned. For language models, the policy is the model and an action is a generated token.
Policy gradient: an example
If a sampled answer earns reward +1 and the average is 0, the update raises the probability of every token in that answer slightly.
Related terms
More LLM training alignment terms
- Post-training
- Preference data
- Prefix tuning
- Pretraining
- Process reward model
- Prompt tuning
- Proximal policy optimisation
- QLoRA
Where you learn it
Policy gradient is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.