AI engineering glossary · Calculus optimisation
What is mini-batch gradient descent?
Mini-batch gradient descent (also called Mini-batch SGD): A compromise that computes each gradient on a small random subset of the data, typically tens to thousands of examples. It reduces noise compared with single-example updates while exploiting GPU parallelism, and is the standard way deep networks are trained.
Mini-batch gradient descent: an example
With 50,000 images and batch size 100, each pass over the data makes 500 parameter updates.
Related terms
More Calculus optimisation terms
- Momentum
- Multivariable chain rule
- Natural logarithm
- Nesterov momentum
- Newton's method
- Numerical differentiation
- Objective function
- Optimisation problem
Where you learn it
Mini-batch gradient descent is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.