AI engineering glossary · Transformers LLM architecture
What is QKV (query, key and value)?
Query, key and value (also called QKV, Q, K, V): The three learned projections of each token used in attention. The query says what a token is looking for, the key advertises what a token contains, and the value is the information actually passed on. A query is compared to all keys, and the resulting weights mix the values.
Query, key and value: an example
For token 'it', the query vector matches strongly with the key of 'animal' and the output is mostly the value vector of 'animal'.
Related terms
More Transformers LLM architecture terms
- Reformer
- Relative position encoding
- Residual stream
- Ring attention
- RoBERTa
- Rotary position embedding
- RWKV
- Scaled dot-product attention
Where you learn it
Query, key and value is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.