AI engineering glossary · Deep learning core
What is weight decay?
Weight decay (also called L2 penalty): A regularisation method that shrinks every weight by a small fraction at each update, discouraging very large weights. In plain SGD it is equivalent to L2 regularisation, which adds a penalty proportional to the squared weights to the loss.
Weight decay: an example
With a decay of 0.01 and learning rate 0.1, each update multiplies weights by roughly 0.999 before adding the gradient step.
Related terms
More Deep learning core terms
- Weight initialisation
- Weight normalisation
- Weight sharing
- Weight tying
- Xavier initialisation
- Activation function
- Adafactor
- Artificial neural network
Where you learn it
Weight decay is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.