AI engineering glossary · LLM training alignment
What is preference data?
Preference data (also called Comparison data, Human preference data): A dataset where each prompt comes with two or more responses and a label saying which one people (or another model) prefer. It is the raw material for training reward models and for direct methods such as DPO. Quality, diversity and annotator agreement determine how good the resulting alignment is.
Preference data: an example
A row holds the prompt 'Suggest a savings plan', a chosen answer that asks about income and goals, and a rejected answer that gives generic advice.
Related terms
More LLM training alignment terms
- Prefix tuning
- Pretraining
- Process reward model
- Prompt tuning
- Proximal policy optimisation
- QLoRA
- Rank (LoRA)
- Reasoning model
Where you learn it
Preference data is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.