AI engineering glossary · LLM training alignment
What is simple preference optimisation?
Simple preference optimisation (also called SimPO): A reference-free preference method that uses the length-normalised average log-probability of a response as its implicit reward and adds a target margin between chosen and rejected answers. Removing the reference model lowers memory use and curbs a tendency toward longer answers.
Simple preference optimisation: an example
Scoring each response by its mean per-token log-probability lets a 20-token and an 80-token answer be compared fairly.
Related terms
More LLM training alignment terms
- Soft labels
- Span corruption
- Superficial alignment hypothesis
- Supervised fine-tuning
- Sycophancy
- Synthetic data
- Task arithmetic
- Teacher-student learning
Where you learn it
Simple preference optimisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.