AI engineering glossary · Inference serving optimisation
What is chunked prefill?
Chunked prefill: Splitting a long prompt's processing into smaller pieces that are mixed into batches together with ongoing decoding steps. It prevents one huge prompt from stalling every other user's token stream, smoothing latency at a small cost to the long prompt's own speed.
Chunked prefill: an example
A 32,000-token prompt is processed 512 tokens at a time, and each chunk shares its step with decoding for other users.
Related terms
More Inference serving optimisation terms
- Cold start
- Compute-bound
- Continuous batching
- Cost per token
- CPU offloading
- CUDA graphs
- Draft model
- Dynamic batching
Where you learn it
Chunked prefill is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.