AI engineering glossary · LLM training alignment
What is goodhart's law?
Goodhart's law (also called Reward over-optimisation): The principle that when a measure becomes a target, it stops being a good measure. In alignment, pushing hard on a proxy such as a reward-model score eventually improves the proxy while true quality gets worse, which is why over-optimisation needs limits.
Goodhart's law: an example
Optimising a chatbot for a reward-model score keeps raising the score, but past a point human raters find the answers worse.
Related terms
More LLM training alignment terms
- Group relative policy optimisation
- Helpful, honest and harmless
- Hugging Face TRL
- Identity preference optimisation
- InstructGPT
- Instruction backtranslation
- Instruction tuning
- Kahneman-Tversky optimisation
Where you learn it
Goodhart's law is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.