AI engineering glossary · Calculus optimisation
What is learning rate warmup?
Learning rate warmup: A schedule phase that starts training with a very small learning rate and raises it gradually over the first steps to its target value. It stabilises early updates for adaptive optimisers and large models, where initial gradient estimates are unreliable.
Learning rate warmup: an example
Increase the rate linearly from 0 to 3e-4 across the first 2,000 steps, then apply cosine decay.
Related terms
More Calculus optimisation terms
- Level set
- Levenberg-Marquardt algorithm
- Limit
- Line search
- Linear approximation
- Linear programming
- Lipschitz continuity
- Local minimum
Where you learn it
Learning rate warmup is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.