AI engineering glossary · Transformers LLM architecture
What is attention head?
Attention head: One of the parallel attention computations inside a multi-head attention layer, with its own query, key and value projection matrices. Each head produces its own pattern of weights, and researchers have found heads that track things like previous tokens, syntax links or repeated text.
Attention head: an example
Llama-style 7B models typically use 32 heads per layer, so a 32-layer model has 1,024 attention heads in total.
Related terms
More Transformers LLM architecture terms
- Attention mechanism
- Attention pattern
- Attention sink
- Attention weights
- Autoregressive generation
- Autoregressive model
- BART
- Beam search
Where you learn it
Attention head is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.