AI engineering glossary · Transformers LLM architecture
What is MQA (multi-query attention)?
Multi-query attention (also called MQA): A variant of multi-head attention in which all query heads share a single set of key and value heads. It shrinks the KV cache and memory traffic dramatically, speeding up generation, with some quality loss compared with full multi-head attention.
Multi-query attention: an example
With 32 query heads, the KV cache is 32 times smaller than standard multi-head attention because only 1 key-value head is stored.
Related terms
More Transformers LLM architecture terms
- Multi-token prediction
- Open-weight model
- Padding mask
- Parameter count
- Position interpolation
- Positional encoding
- Post-norm
- Pre-norm
Where you learn it
Multi-query attention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.