AI engineering glossary · Inference serving optimisation
What is prefix caching?
Prefix caching (also called Automatic prefix caching): Reusing the stored KV cache for a prompt beginning that has been seen before, so the model does not recompute it. It cuts time to first token and cost when many requests share a long system prompt, document or conversation history.
Prefix caching: an example
A support bot with a 3,000-token policy prompt computes it once; every later request reuses its cache and only processes the customer's new question.
Related terms
More Inference serving optimisation terms
- Prompt lookup decoding
- Pruning
- Quantisation
- Quantisation-aware training
- Queueing delay
- RadixAttention
- Request scheduler
- Reserved capacity
Where you learn it
Prefix caching is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.