AI engineering glossary · LLM training alignment
What is policy (reinforcement learning)?
Policy (reinforcement learning): The rule an agent uses to choose actions given a state. In language-model reinforcement learning, the policy is the model itself: the state is the prompt plus text generated so far, and the action is the next token, sampled from the model's probability distribution.
Policy (reinforcement learning): an example
When a chat model is trained with RLHF, the model being updated is called the policy, while a frozen copy is the reference model.
Related terms
More LLM training alignment terms
- Policy gradient
- Post-training
- Preference data
- Prefix tuning
- Pretraining
- Process reward model
- Prompt tuning
- Proximal policy optimisation
Where you learn it
Policy (reinforcement learning) is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.