AI engineering glossary · Calculus optimisation
What is momentum?
Momentum (also called Heavy-ball method, SGD with momentum): An optimisation trick that keeps a running velocity, an exponentially weighted sum of past gradients, and updates parameters using it. Steps in consistent directions build speed while oscillating directions cancel, which accelerates progress along narrow valleys.
Momentum: an example
With momentum 0.9, velocity = 0.9 * velocity + gradient, then weights change by learning rate times velocity.
Related terms
More Calculus optimisation terms
- Multivariable chain rule
- Natural logarithm
- Nesterov momentum
- Newton's method
- Numerical differentiation
- Objective function
- Optimisation problem
- Partial derivative
Where you learn it
Momentum is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.