AI engineering glossary · Inference serving optimisation
What is prefill-decode disaggregation?
Prefill-decode disaggregation (also called Disaggregated serving): Running prompt processing and token generation on separate groups of GPUs. Prefill is compute-heavy and decode is memory-heavy, so separating them lets each be tuned and scaled independently and avoids interference, at the cost of transferring the KV cache between machines.
Prefill-decode disaggregation: an example
Four GPUs handle prompts and eight handle generation, with the cache sent over a fast interconnect after each prompt.
Related terms
More Inference serving optimisation terms
- Prefix caching
- Prompt lookup decoding
- Pruning
- Quantisation
- Quantisation-aware training
- Queueing delay
- RadixAttention
- Request scheduler
Where you learn it
Prefill-decode disaggregation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.