AI engineering glossary · LLM training alignment
What is alignment tax?
Alignment tax: The loss in performance on some capabilities that can occur after alignment training, such as lower benchmark scores after RLHF. Techniques like mixing in pretraining data or careful regularisation aim to reduce it.
Alignment tax: an example
After RLHF, a model becomes more polite but its score on a reading-comprehension benchmark falls by a few points.
Related terms
More LLM training alignment terms
- Alpaca
- Base model
- Bradley-Terry model
- Causal language modelling
- Chat template
- Chinchilla scaling
- Chosen and rejected responses
- Clipped surrogate objective
Where you learn it
Alignment tax is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.