AI engineering glossary · LLM training alignment
What is DPO (direct preference optimisation)?
Direct preference optimisation (also called DPO): An alignment method that trains a model directly on chosen-versus-rejected pairs with a simple classification-style loss, removing the separate reward model and the reinforcement learning loop. It raises the relative likelihood of chosen answers compared with a frozen reference model, and is cheaper and steadier than PPO.
Direct preference optimisation: an example
loss = -log sigmoid(beta * [(log pi(chosen) - log ref(chosen)) - (log pi(rejected) - log ref(rejected))]), with beta commonly 0.1.
Related terms
More LLM training alignment terms
- Domain-adaptive pretraining
- DoRA
- Double quantisation
- Emergent abilities
- Evol-Instruct
- Exposure bias
- FLAN
- FLOPs
Where you learn it
Direct preference optimisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.