AI engineering glossary · Calculus optimisation
What is gradient checking?
Gradient checking: A debugging test that compares an analytically or automatically computed gradient with a finite-difference estimate. A large relative gap suggests a bug in the gradient code. It is slow, so it is used on small examples rather than during real training.
Gradient checking: an example
If autodiff gives 6.000 and finite differences give 6.001, the relative error is about 0.00008, so the gradient passes the check.
Related terms
More Calculus optimisation terms
- Gradient clipping
- Gradient descent
- Hessian matrix
- Integral
- Jacobian matrix
- Jacobian-vector product
- KKT conditions
- L-BFGS
Where you learn it
Gradient checking is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.