AI engineering glossary · Calculus optimisation
What is GD (gradient descent)?
Gradient descent (also called GD, Steepest descent): An iterative algorithm that minimises a function by repeatedly moving its variables a small step in the direction opposite the gradient, the direction of steepest decrease. It is the basis of how nearly every machine learning model is trained.
Gradient descent: an example
Minimise f(w) = (w - 3)^2 from w = 0 with learning rate 0.1: the update w - 0.1 * 2(w - 3) moves w to 0.6, then 1.08, then 1.46, approaching 3.
Gradient descent in Python
A short example you can run (Python 3.11). The output below is what it really prints.
# Minimise f(w) = (w - 3)^2, whose gradient is 2(w - 3)
w, lr = 0.0, 0.1
for step in range(1, 26):
grad = 2 * (w - 3)
w -= lr * grad
if step % 5 == 0:
print(f"step {step:2d} w = {w:.4f} loss = {(w - 3) ** 2:.6f}")Output
step 5 w = 2.0170 loss = 0.966368 step 10 w = 2.6779 loss = 0.103763 step 15 w = 2.8944 loss = 0.011141 step 20 w = 2.9654 loss = 0.001196 step 25 w = 2.9887 loss = 0.000128
Related terms
More Calculus optimisation terms
- Hessian matrix
- Integral
- Jacobian matrix
- Jacobian-vector product
- KKT conditions
- L-BFGS
- L-smoothness
- Lagrange multiplier
Where you learn it
Gradient descent is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.