AI engineering glossary · Calculus optimisation
What is numerical differentiation?
Numerical differentiation: Estimating derivatives by evaluating a function at nearby inputs rather than deriving a formula. It works for any function you can run, but it needs one or more evaluations per input dimension and is sensitive to the step size chosen.
Numerical differentiation: an example
For a model with 1,000 parameters, estimating the full gradient this way needs at least 1,001 loss evaluations, which is why training uses automatic differentiation instead.
Related terms
More Calculus optimisation terms
- Objective function
- Optimisation problem
- Partial derivative
- Power rule
- Preconditioning
- Product rule
- Projected gradient descent
- Proximal gradient method
Where you learn it
Numerical differentiation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.