AI engineering glossary · Transformers LLM architecture
What is pre-norm?
Pre-norm (also called Pre-LN): A block layout where normalisation is applied to the input of each sublayer, before attention or the MLP, and the sublayer output is added back to the unnormalised residual path. It makes very deep models much more stable to train than the original arrangement and is standard in modern LLMs.
Pre-norm: an example
x = x + Attention(LayerNorm(x)); x = x + MLP(LayerNorm(x)).
Related terms
More Transformers LLM architecture terms
- Prefill and decode
- Prefix language model
- QK-norm
- Quadratic attention cost
- Query, key and value
- Reformer
- Relative position encoding
- Residual stream
Where you learn it
Pre-norm is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.