AI engineering glossary · Transformers LLM architecture
What is causal masking?
Causal masking (also called Causal attention, Masked self-attention, Look-ahead mask): Blocking each position in attention from seeing later positions, done by setting scores for future tokens to negative infinity before softmax so their weights become zero. It ensures a model trained to predict the next token cannot cheat by looking ahead, and is what makes decoder-only models generative.
Causal masking: an example
For a 4-token input, token 2 can attend only to tokens 1 and 2; the score matrix is lower-triangular after masking.
Related terms
More Transformers LLM architecture terms
- Constrained decoding
- Context window
- Contrastive decoding
- Cross-attention
- Decoder
- Decoder-only model
- Decoding
- Dense model
Where you learn it
Causal masking is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.