AI engineering glossary · Inference serving optimisation
What is semantic caching?
Semantic caching: Storing past answers and returning one when a new query means the same as an earlier query, judged by embedding similarity rather than exact text. It saves whole model calls, but risks returning a wrong answer if similarity is judged too loosely.
Semantic caching: an example
'How do I reset my password?' and 'password reset steps' hit the same stored answer when similarity exceeds a chosen threshold.
Related terms
- Prompt caching
- Embedding similarity
- Cache hit
- Cost per token
More Inference serving optimisation terms
- Semi-structured sparsity
- Service level objective
- SGLang
- Shared memory
- Small language model
- SmoothQuant
- Sparsity
- Speculative decoding
Where you learn it
Semantic caching is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.