AI engineering glossary · Calculus optimisation
What is reverse-mode automatic differentiation?
Reverse-mode automatic differentiation (also called Reverse mode): A variant that first runs the computation forward, then sweeps backward from the output propagating sensitivities to every input, giving the whole gradient of a scalar output in one backward pass. It is efficient for losses with millions of parameters and is the basis of backpropagation.
Reverse-mode automatic differentiation: an example
For a loss over 100 million parameters, one forward pass plus one backward pass yields all 100 million partial derivatives, at a cost of a few forward passes.
Related terms
- Forward-mode automatic differentiation
- Vector-Jacobian product
- Computational graph
- Automatic differentiation
More Calculus optimisation terms
- Riemann sum
- RMSProp
- Saddle point
- Secant line
- Second-order approximation
- Second-order condition
- Sharpness-aware minimisation
- Simulated annealing
Where you learn it
Reverse-mode automatic differentiation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.