AI engineering glossary · Transformers LLM architecture
What is head dimension?
Head dimension (also called d_head, d_k): The size of the query, key and value vectors in each attention head, usually the model dimension divided by the number of heads. Common values are 64 and 128. It affects the shape of the KV cache and the efficiency of attention kernels.
Head dimension: an example
4,096 model dimension with 32 heads gives a head dimension of 128.
Related terms
More Transformers LLM architecture terms
- Hidden states
- Induction head
- Instruct model
- KV cache
- Large language model
- Learned positional embedding
- Length penalty
- Linear attention
Where you learn it
Head dimension is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.