AI engineering glossary · Transformers LLM architecture
What is decoding?
Decoding (also called Text generation strategy): The process of turning a language model's next-token probabilities into actual output text by choosing a token at each step under some strategy, such as always taking the most likely one or sampling. The strategy strongly affects how creative, repetitive or accurate the output is.
Decoding: an example
The same prompt gives a fixed answer with greedy decoding but varied answers when sampling at temperature 1.0.
Related terms
More Transformers LLM architecture terms
- Dense model
- Diverse beam search
- ELECTRA
- Encoder
- Encoder-decoder model
- Encoder-only model
- Expert
- Expert parallelism
Where you learn it
Decoding is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.