AI engineering glossary · Transformers LLM architecture
What is QK-norm?
QK-norm: Applying normalisation to the query and key vectors before their dot product in attention. It keeps attention scores from growing too large, which stabilises training of big models and helps avoid loss spikes caused by saturated softmax.
QK-norm: an example
Each head's query and key vectors are normalised to controlled scale before computing Q K^T.
Related terms
More Transformers LLM architecture terms
- Quadratic attention cost
- Query, key and value
- Reformer
- Relative position encoding
- Residual stream
- Ring attention
- RoBERTa
- Rotary position embedding
Where you learn it
QK-norm is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.