AI engineering glossary · Calculus optimisation
What is SGDR (warm restarts)?
Warm restarts (also called SGDR): A schedule technique that periodically resets the learning rate to a high value and decays it again, often with cosine annealing, so training can leave one basin and explore others. Each cycle can end with a good candidate solution.
Warm restarts: an example
A cosine schedule of 10 epochs is repeated three times, each cycle starting again from lr = 0.1 and decaying to near zero.
Related terms
More Calculus optimisation terms
- AdaGrad
- Adam
- AdamW
- Antiderivative
- Armijo condition
- Automatic differentiation
- Backtracking line search
- Batch gradient descent
Where you learn it
Warm restarts is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.