AI engineering glossary · LLM training alignment
What is reward model?
Reward model (also called Preference model): A model that reads a prompt and a response and outputs a single score estimating how much humans would prefer it. It is usually a language model with a scalar output head, trained on ranked pairs of responses. It supplies the training signal in RLHF and can also rank candidate answers.
Reward model: an example
Given two replies to 'Explain EMI', the reward model scores the clearer one 2.1 and the vague one -0.4, matching the human ranking.
Related terms
More LLM training alignment terms
- Safety fine-tuning
- Scalable oversight
- Scaling laws
- Self-instruct
- Self-rewarding language models
- Self-Taught Reasoner
- Sequence packing
- Sequence-level distillation
Where you learn it
Reward model is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.