AI engineering glossary · Deep learning core
What is weight initialisation?
Weight initialisation (also called Parameter initialisation): The choice of starting values for a network's weights before training. Good starting values keep signals and gradients at a workable scale from layer to layer; poor ones cause vanishing or exploding gradients or leave neurons identical.
Weight initialisation: an example
Drawing weights from a normal distribution with standard deviation 0.01 in a 50-layer network would shrink signals to nothing.
Related terms
More Deep learning core terms
- Weight normalisation
- Weight sharing
- Weight tying
- Xavier initialisation
- Activation function
- Adafactor
- Artificial neural network
- Autoencoder
Where you learn it
Weight initialisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.