AI engineering glossary · Deep learning core
What is layer normalisation?
Layer normalisation (also called LayerNorm): A layer that standardises the values across the features of a single example rather than across the batch, then applies a learned scale and shift. It works with any batch size, which is why transformers and recurrent networks use it.
Layer normalisation: an example
Each token vector of 768 numbers in a transformer is normalised using its own mean and variance.
Related terms
More Deep learning core terms
- Leaky ReLU
- Learning rate finder
- LeCun initialisation
- Linear scaling rule
- Lion optimiser
- Log-sum-exp trick
- Logits
- Loss scaling
Where you learn it
Layer normalisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.