AI engineering glossary · LLM training alignment
What is loss masking?
Loss masking (also called Prompt masking, Completion-only loss): Excluding chosen tokens from the training loss so that gradients come only from the parts you want the model to learn. In supervised fine-tuning the prompt tokens are usually masked, so the model is scored only on the assistant's reply, not on reproducing the user's question.
Loss masking: an example
For a 30-token question and a 50-token answer, only the 50 answer tokens contribute to the loss; the 30 question tokens are given label -100 in common libraries.
Related terms
More LLM training alignment terms
- Loss spike
- Masked language modelling
- Maximal update parametrisation
- Megatron-LM
- Model collapse
- Model FLOPs utilisation
- Model merging
- Next-token prediction
Where you learn it
Loss masking is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.