AI engineering glossary · Transformers LLM architecture
What is prefill and decode?
Prefill and decode (also called Prompt processing): The two phases of LLM inference. In prefill, the model processes the entire prompt in parallel and fills the KV cache; in decode, it generates output tokens one at a time. Prefill is compute-heavy, while decode is limited mostly by memory bandwidth.
Prefill and decode: an example
A 2,000-token prompt is handled in one prefill pass, followed by 300 decode steps to write a 300-token answer.
Related terms
More Transformers LLM architecture terms
- Prefix language model
- QK-norm
- Quadratic attention cost
- Query, key and value
- Reformer
- Relative position encoding
- Residual stream
- Ring attention
Where you learn it
Prefill and decode is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.