AI engineering glossary · Inference serving optimisation
What is KV cache quantisation?
KV cache quantisation: Storing the cached keys and values in lower precision such as 8-bit or 4-bit. Because the cache grows with context length and batch size, shrinking it lets a GPU serve longer prompts or more users, with a small accuracy risk.
KV cache quantisation: an example
Moving the cache from FP16 to FP8 halves its memory, so a server that held 20 concurrent 8K-token chats can hold about 40.
Related terms
More Inference serving optimisation terms
- llama.cpp
- LLM.int8()
- Load balancing
- Local inference
- Low-rank factorisation
- Medusa decoding
- Memory fragmentation
- Memory-bound
Where you learn it
KV cache quantisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.