AI engineering glossary · LLM training alignment
What is CLM (causal language modelling)?
Causal language modelling (also called Causal LM, CLM): A training objective for decoder-style models in which each position may only attend to earlier positions, so the model learns to predict the next token from the left context alone. A causal mask inside attention enforces this rule. It is the objective behind GPT-style and most modern chat models.
Causal language modelling: an example
For the sentence 'Ram ate rice', the model predicts 'ate' from 'Ram' and 'rice' from 'Ram ate'; the mask stops position 2 from seeing position 3 during training.
Related terms
More LLM training alignment terms
- Chat template
- Chinchilla scaling
- Chosen and rejected responses
- Clipped surrogate objective
- Common Crawl
- Compute-optimal training
- Constitutional AI
- Continued pretraining
Where you learn it
Causal language modelling is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.