Gradient descent from scratch in Python

Gradient descent is how almost every model learns: measure how wrong it is, work out which way is downhill, take a small step, repeat. Here you write it yourself, in plain Python, and run it on this page.

Python 3.12 in your browser (Pyodide)Term: Gradient descent

Step 1: minimise a simple curve

f(w) = (w − 3)² is smallest at w = 3. Its gradient (slope) is 2(w − 3). Each step moves w a little against the slope.

Code for Step 1: minimise a simple curve

w = 0.0          # start anywhere
lr = 0.1         # learning rate: the step size
for step in range(1, 31):
    grad = 2 * (w - 3)
    w = w - lr * grad
    if step % 5 == 0:
        print(f"step {step:2d}: w = {w:.4f}")

Step 2: fit a line to data

The same idea fits a model: find the slope m and intercept b that make predictions closest to the data (mean squared error).

Code for Step 2: fit a line to data

import numpy as np
x = np.array([1, 2, 3, 4, 5], dtype=float)
y = np.array([3.1, 4.9, 7.2, 8.8, 11.1])   # roughly y = 2x + 1
m, b, lr = 0.0, 0.0, 0.02
for epoch in range(2000):
    pred = m * x + b
    dm = (2 / len(x)) * np.sum((pred - y) * x)
    db = (2 / len(x)) * np.sum(pred - y)
    m, b = m - lr * dm, b - lr * db
print(f"m = {m:.3f}, b = {b:.3f}")
print("loss:", round(float(np.mean((m * x + b - y) ** 2)), 4))

Step 3: break it on purpose

Change lr to 1.1 in step 1 and run it again. The steps overshoot and w runs away: a learning rate that is too big never settles.