AI engineering glossary · Inference serving optimisation
What is KV cache eviction?
KV cache eviction: Choosing which stored keys and values to drop when cache memory is full. Policies range from removing whole least-recently-used requests to discarding individual tokens judged unimportant, trading memory saved against risk of losing needed context.
KV cache eviction: an example
A long chat keeps its first few tokens and latest 2,000 tokens in cache while dropping the middle.
Related terms
More Inference serving optimisation terms
- KV cache quantisation
- llama.cpp
- LLM.int8()
- Load balancing
- Local inference
- Low-rank factorisation
- Medusa decoding
- Memory fragmentation
Where you learn it
KV cache eviction is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.