AI engineering glossary · Transformers LLM architecture
What is MHA (multi-head attention)?
Multi-head attention (also called MHA): Running several attention operations in parallel, each with its own learned query, key and value projections over a smaller dimension, then concatenating their outputs and mixing them with one more linear layer. Different heads can specialise in different relationships, such as syntax or nearby words.
Multi-head attention: an example
A model with d_model = 768 and 12 heads gives each head 64 dimensions; the 12 outputs are concatenated back to 768.
Related terms
More Transformers LLM architecture terms
- Multi-head latent attention
- Multi-query attention
- Multi-token prediction
- Open-weight model
- Padding mask
- Parameter count
- Position interpolation
- Positional encoding
Where you learn it
Multi-head attention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.