AI engineering glossary · Deep learning core
What is RMSNorm?
RMSNorm (also called Root mean square layer normalisation): A simplified layer normalisation that rescales a vector by its root mean square and a learned gain, skipping the mean subtraction. It is slightly cheaper than full layer normalisation and is used in several recent large language models.
RMSNorm: an example
For a vector [3, 4], the root mean square is sqrt((9+16)/2), about 3.54, so the output is about [0.85, 1.13] before gain.
Related terms
More Deep learning core terms
- Saturation
- SELU
- SiLU
- Skip connection
- Softmax
- Softplus
- Sparse categorical cross-entropy
- Spectral normalisation
Where you learn it
RMSNorm is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.