AI engineering glossary · Calculus optimisation
What is AD (automatic differentiation)?
Automatic differentiation (also called Autodiff, AD): A technique that computes exact derivatives of a program by recording elementary operations and applying the chain rule to each of them, without deriving formulas by hand or using approximations. It is what PyTorch, TensorFlow and JAX use to obtain gradients of losses.
Automatic differentiation: an example
In PyTorch, setting x = torch.tensor(3.0, requires_grad=True), y = x**2 and calling y.backward() leaves x.grad equal to 6.0.
Related terms
- Reverse-mode automatic differentiation
- Forward-mode automatic differentiation
- Computational graph
- Chain rule
More Calculus optimisation terms
- Backtracking line search
- Batch gradient descent
- Bayesian optimisation
- BFGS
- Chain rule
- Computational graph
- Condition number
- Conjugate gradient method
Where you learn it
Automatic differentiation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.