AI engineering glossary · Calculus optimisation
What is variance reduction?
Variance reduction: A family of techniques such as SVRG and SAGA that lower the noise of stochastic gradient estimates by using stored or periodically computed full gradients as a reference. They allow larger steps and faster, linear convergence on convex finite-sum problems.
Variance reduction: an example
SVRG recomputes the full-data gradient once per epoch and corrects each mini-batch gradient using the difference from that snapshot.
Related terms
More Calculus optimisation terms
- Vector-Jacobian product
- Warm restarts
- AdaGrad
- Adam
- AdamW
- Antiderivative
- Armijo condition
- Automatic differentiation
Where you learn it
Variance reduction is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.