AI engineering glossary · Transformers LLM architecture
What is KV cache?
KV cache (also called Key-value cache): Memory that stores the key and value vectors of every token already processed at each layer, so that during generation each new token only computes its own keys and values instead of recomputing the whole prompt. It speeds up decoding but its size grows with sequence length, layers and batch.
KV cache: an example
Without it, generating token 501 would reprocess all 500 earlier tokens; with it, only the new token is computed and attends to cached keys and values.
Related terms
More Transformers LLM architecture terms
- Large language model
- Learned positional embedding
- Length penalty
- Linear attention
- Llama
- LM head
- Load-balancing loss
- Log probabilities
Where you learn it
KV cache is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.