AI engineering glossary · Transformers LLM architecture
What is constrained decoding?
Constrained decoding (also called Grammar-constrained decoding, Structured decoding): Decoding in which allowed tokens are restricted at each step, by masking logits, so the output must follow a grammar, schema or pattern such as valid JSON. It guarantees the format at the cost of some extra computation.
Constrained decoding: an example
To get valid JSON, the decoder masks every token that would break the schema, so after '{"age":' only digits are allowed.
Related terms
More Transformers LLM architecture terms
- Context window
- Contrastive decoding
- Cross-attention
- Decoder
- Decoder-only model
- Decoding
- Dense model
- Diverse beam search
Where you learn it
Constrained decoding is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.