AI engineering glossary · Calculus optimisation
What is learning rate?
Learning rate (also called Step size, Alpha): The scalar that sets how large a step an optimiser takes along the negative gradient. Too small and training crawls; too large and it overshoots or diverges. It is usually the most important setting to tune.
Learning rate: an example
With gradient 4 and learning rate 0.01, a weight changes by 0.04 per step; with learning rate 1.0 it would jump by 4 and could bounce past the minimum.
Related terms
More Calculus optimisation terms
- Learning rate schedule
- Learning rate warmup
- Level set
- Levenberg-Marquardt algorithm
- Limit
- Line search
- Linear approximation
- Linear programming
Where you learn it
Learning rate is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.