AI engineering glossary · Deep learning core
What is weight normalisation?
Weight normalisation: A reparameterisation that writes each weight vector as a direction times a separate learnable length, w = g * v / ||v||. Decoupling scale from direction can speed up optimisation without depending on batch statistics.
Weight normalisation: an example
A neuron's weight vector is stored as a unit direction v/||v|| and a scalar g learned independently.
Related terms
More Deep learning core terms
- Weight sharing
- Weight tying
- Xavier initialisation
- Activation function
- Adafactor
- Artificial neural network
- Autoencoder
- Autograd
Where you learn it
Weight normalisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.