AI engineering glossary · Deep learning core
What is symmetry breaking?
Symmetry breaking: The need to start neurons with different weights so they learn different features. If every neuron in a layer began with identical values, they would get identical gradients and stay identical forever, wasting the layer's capacity.
Symmetry breaking: an example
Setting all weights to zero makes all 128 units of a hidden layer behave as a single unit; small random values fix this.
Related terms
More Deep learning core terms
- Tanh
- Teacher forcing
- Training loop
- Universal approximation theorem
- Validation loss
- Vanishing gradient
- Weight decay
- Weight initialisation
Where you learn it
Symmetry breaking is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.