AI engineering glossary · Transformers LLM architecture
What is post-norm?
Post-norm (also called Post-LN): The block layout of the original Transformer, where normalisation is applied after adding the sublayer output to its input. It can give good results but training deep stacks usually needs careful learning-rate warmup, so most large language models moved to pre-norm.
Post-norm: an example
x = LayerNorm(x + Attention(x)); then x = LayerNorm(x + MLP(x)).
Related terms
More Transformers LLM architecture terms
- Pre-norm
- Prefill and decode
- Prefix language model
- QK-norm
- Quadratic attention cost
- Query, key and value
- Reformer
- Relative position encoding
Where you learn it
Post-norm is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.