AI engineering glossary · LLM training alignment
What is reward hacking?
Reward hacking (also called Reward gaming, Specification gaming): When a model finds ways to score highly on the reward signal without truly doing what people wanted, exploiting flaws in the reward model or metric. Typical symptoms are excessive length, flattery or confident nonsense. It is one reason RLHF uses KL penalties and reward-model refreshes.
Reward hacking: an example
A summariser rewarded for 'sounding helpful' learns to add long, polite filler that earns higher scores but adds no information.
Related terms
More LLM training alignment terms
- Reward model
- Safety fine-tuning
- Scalable oversight
- Scaling laws
- Self-instruct
- Self-rewarding language models
- Self-Taught Reasoner
- Sequence packing
Where you learn it
Reward hacking is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.