AI engineering glossary · Transformers LLM architecture
What is GQA (grouped-query attention)?
Grouped-query attention (also called GQA): A compromise between multi-head and multi-query attention: query heads are split into groups and each group shares one key and value head. It keeps most of the quality of full attention while cutting KV cache size, and is common in current open models.
Grouped-query attention: an example
With 32 query heads and 8 key-value heads, each key-value head serves 4 query heads and the cache is 4 times smaller.
Related terms
More Transformers LLM architecture terms
- Head dimension
- Hidden states
- Induction head
- Instruct model
- KV cache
- Large language model
- Learned positional embedding
- Length penalty
Where you learn it
Grouped-query attention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.