AI engineering glossary · Deep learning core
What is SWA (stochastic weight averaging)?
Stochastic weight averaging (also called SWA): A technique that averages the weights from several points late in training, typically taken with a constant or cyclical learning rate. The averaged model tends to sit in a flatter region and generalise better.
Stochastic weight averaging: an example
Averaging weights saved at the end of each of the last 10 epochs gives one final model.
Related terms
More Deep learning core terms
- Straight-through estimator
- SwiGLU
- Symmetry breaking
- Tanh
- Teacher forcing
- Training loop
- Universal approximation theorem
- Validation loss
Where you learn it
Stochastic weight averaging is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.