AI engineering glossary · Inference serving optimisation
What is preemption?
Preemption: Pausing a running request so its resources can be given to others, usually when GPU memory for the KV cache runs out. The paused request's cache is either swapped to CPU memory or discarded and recomputed later, costing extra time.
Preemption: an example
When memory fills up, the engine evicts the newest request's cache and re-runs its prompt once space frees.
Related terms
More Inference serving optimisation terms
- Prefill-decode disaggregation
- Prefix caching
- Prompt lookup decoding
- Pruning
- Quantisation
- Quantisation-aware training
- Queueing delay
- RadixAttention
Where you learn it
Preemption is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.