AI engineering glossary · Calculus optimisation
What is gradient accumulation?
Gradient accumulation: A technique that sums gradients from several small mini-batches before making one parameter update, imitating a larger batch when memory is limited. The effective batch size is the micro-batch size times the number of accumulation steps.
Gradient accumulation: an example
A GPU that fits 8 examples at a time can run 4 micro-batches and then update once, giving an effective batch size of 32.
Related terms
More Calculus optimisation terms
- Gradient checking
- Gradient clipping
- Gradient descent
- Hessian matrix
- Integral
- Jacobian matrix
- Jacobian-vector product
- KKT conditions
Where you learn it
Gradient accumulation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.