AI engineering glossary · Calculus optimisation
What is newton's method?
Newton's method (also called Newton-Raphson method): An iterative optimiser that uses both the gradient and Hessian to model the objective as a quadratic and jump straight to that quadratic's minimum: x_new = x - H^-1 * g. It converges very fast near a solution but each step is costly for many parameters.
Newton's method: an example
For f(x) = x^2 - 4x, at x = 0 the gradient is -4 and the Hessian is 2, so one step gives x = 0 - (-4/2) = 2, the exact minimum.
Related terms
More Calculus optimisation terms
- Numerical differentiation
- Objective function
- Optimisation problem
- Partial derivative
- Power rule
- Preconditioning
- Product rule
- Projected gradient descent
Where you learn it
Newton's method is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.