AI engineering glossary · LLM training alignment
What is ORPO (odds ratio preference optimisation)?
Odds ratio preference optimisation (also called ORPO): A method that merges supervised fine-tuning and preference alignment into one stage by adding an odds-ratio penalty that favours chosen over rejected responses to the ordinary language-modelling loss. It needs no reference model, saving memory.
Odds ratio preference optimisation: an example
Instead of running SFT and then DPO, one training run on chosen-and-rejected pairs applies both losses together.
Related terms
More LLM training alignment terms
- Overtrained model
- Paged optimisers
- Parameter-efficient fine-tuning
- PEFT library
- Pipeline parallelism
- Policy (reinforcement learning)
- Policy gradient
- Post-training
Where you learn it
Odds ratio preference optimisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.