AI engineering glossary · Calculus optimisation
What is cosine annealing?
Cosine annealing: A learning-rate schedule that lowers the rate from an initial value to a minimum following half a cosine curve, decreasing slowly at first, faster in the middle and gently at the end. It is widely used for training vision and language models.
Cosine annealing: an example
lr(t) = lr_min + 0.5 * (lr_max - lr_min) * (1 + cos(pi * t / T)); with lr_max = 0.1 and lr_min = 0 at halfway, lr = 0.05.
Related terms
More Calculus optimisation terms
- Derivative
- Derivative-free optimisation
- Differentiability
- Directional derivative
- Duality
- Euler's number
- Exponential function
- Feasible region
Where you learn it
Cosine annealing is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.