AI engineering glossary · Calculus optimisation
What is SGD (stochastic gradient descent)?
Stochastic gradient descent (also called SGD): A variant that updates parameters using the gradient computed from a single randomly chosen training example (or, loosely, a small random batch). Updates are cheap and noisy; the noise can help escape shallow traps and makes learning on huge datasets feasible.
Stochastic gradient descent: an example
For linear regression on 1 million rows, pick one row, compute its error gradient, adjust the weights, and repeat with another random row.
Related terms
More Calculus optimisation terms
- Strong convexity
- Subgradient
- Subgradient method
- Sum rule
- Symbolic differentiation
- Tangent line
- Taylor series
- Total derivative
Where you learn it
Stochastic gradient descent is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.