AI engineering glossary · LLM training alignment
What is KL penalty?
KL penalty (also called KL regularisation, KL divergence penalty): A term added to the RLHF reward that subtracts a multiple of the KL divergence between the tuned model and a frozen reference model. It stops the policy from drifting into odd text that games the reward model, and preserves fluency and knowledge from earlier training.
KL penalty: an example
reward_total = reward_model_score - beta * KL(policy || reference), with beta = 0.05 penalising answers whose token probabilities depart far from the reference.
Related terms
More LLM training alignment terms
- Knowledge distillation
- LIMA
- Long-context fine-tuning
- LoRA
- Loss masking
- Loss spike
- Masked language modelling
- Maximal update parametrisation
Where you learn it
KL penalty is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.