AI engineering glossary · Inference serving optimisation
What is cache-aware routing?
Cache-aware routing (also called Prefix-aware routing): A load-balancing policy that sends a request to the replica most likely to already hold a matching prompt prefix in its KV cache, so prefill work is reused instead of repeated. It must be balanced against keeping load even.
Cache-aware routing: an example
All requests starting with the same 2,000-token system prompt are steered to the same two replicas.
Related terms
More Inference serving optimisation terms
- Calibration dataset
- Chunked prefill
- Cold start
- Compute-bound
- Continuous batching
- Cost per token
- CPU offloading
- CUDA graphs
Where you learn it
Cache-aware routing is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.