AI engineering glossary · Inference serving optimisation
What is radixAttention?
RadixAttention: A technique in the SGLang serving system that keeps KV caches in a radix tree keyed by token sequence, so requests with common prefixes automatically share cached computation and least-recently-used branches are evicted when memory is short.
RadixAttention: an example
Two agent runs that share the same tool instructions and first three turns reuse the same tree branch.
Related terms
More Inference serving optimisation terms
- Request scheduler
- Reserved capacity
- Roofline model
- Scale factor
- Semantic caching
- Semi-structured sparsity
- Service level objective
- SGLang
Where you learn it
RadixAttention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.