AI engineering glossary · Transformers LLM architecture
What is padding mask?
Padding mask (also called Attention mask): A mask that stops attention from using padding tokens, the filler added to shorter sequences so a batch has equal length. Masked positions get negative-infinity scores so they receive zero weight and do not affect real tokens or the loss.
Padding mask: an example
Batching sentences of length 5 and 8 pads the first with 3 filler tokens; the padding mask marks those 3 positions as ignored.
Related terms
More Transformers LLM architecture terms
- Parameter count
- Position interpolation
- Positional encoding
- Post-norm
- Pre-norm
- Prefill and decode
- Prefix language model
- QK-norm
Where you learn it
Padding mask is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.