AI engineering glossary · Calculus optimisation
What is NAG (nesterov momentum)?
Nesterov momentum (also called Nesterov accelerated gradient, NAG): A refinement of momentum that evaluates the gradient at the look-ahead point where the current velocity would take the parameters, rather than at the current point. This early correction reduces overshoot and gives better theoretical rates for convex problems.
Nesterov momentum: an example
With velocity 0.5 and weight 2.0, it computes the gradient at about 2.0 + 0.9 * 0.5 rather than at 2.0.
Related terms
More Calculus optimisation terms
- Newton's method
- Numerical differentiation
- Objective function
- Optimisation problem
- Partial derivative
- Power rule
- Preconditioning
- Product rule
Where you learn it
Nesterov momentum is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.