AI engineering glossary · LLM training alignment
What is MLM (masked language modelling)?
Masked language modelling (also called Masked LM, MLM): A training objective where some tokens in the input are hidden and the model must recover them using context from both the left and the right. It suits encoder models that build rich representations of a sentence, and was popularised by BERT. Unlike next-token prediction, it does not generate text left to right.
Masked language modelling: an example
From 'UPI payments are [MASK] in India', a model fills the blank with 'popular' by using words on both sides.
Related terms
More LLM training alignment terms
- Maximal update parametrisation
- Megatron-LM
- Model collapse
- Model FLOPs utilisation
- Model merging
- Next-token prediction
- NormalFloat 4-bit
- Odds ratio preference optimisation
Where you learn it
Masked language modelling is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.