AI engineering glossary · Deep learning core
What is exponential moving average of weights?
Exponential moving average of weights (also called Weight EMA): A smoothed copy of a model's weights updated after each step as a weighted average, ema = decay*ema + (1 - decay)*weights. Evaluating with the averaged weights often gives steadier and slightly better results than the raw final weights.
Exponential moving average of weights: an example
With decay 0.999, the average reflects roughly the last 1,000 updates.
Related terms
More Deep learning core terms
- Fan-in and fan-out
- Feedforward neural network
- Flat minima
- Focal loss
- Forward pass
- Fully connected layer
- Gated linear unit
- Gating mechanism
Where you learn it
Exponential moving average of weights is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.