AI engineering glossary · Inference serving optimisation
What is pagedAttention?
PagedAttention: A technique that stores a request's KV cache in fixed-size blocks that need not sit side by side in GPU memory, tracked with a table like an operating system's page table. It removes fragmentation and waste, letting a server hold many more concurrent requests.
PagedAttention: an example
With 16-token blocks, a 300-token conversation uses 19 blocks that can be scattered anywhere in memory.
Related terms
More Inference serving optimisation terms
- Per-channel quantisation
- Per-tensor quantisation
- Percentile latency
- Post-training quantisation
- Preemption
- Prefill-decode disaggregation
- Prefix caching
- Prompt lookup decoding
Where you learn it
PagedAttention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.