AI engineering glossary · Deep learning core
What is backpropagation?
Backpropagation (also called Backprop, Backward propagation): The algorithm that computes how much each weight contributed to the error. It applies the chain rule of calculus backward from the loss through every layer, producing a gradient for each parameter, which an optimiser then uses to update the weights.
Backpropagation: an example
If raising a weight by 0.01 would raise the loss by 0.002, backpropagation reports a gradient near 0.2 for that weight.
Backpropagation in Python
A short example you can run (Python 3.11). The output below is what it really prints.
# One neuron: y = w*x + b, loss = (y - target)^2. Chain rule by hand.
x, target = 2.0, 10.0
w, b, lr = 1.0, 0.0, 0.05
for step in range(3):
y = w * x + b
loss = (y - target) ** 2
dloss_dy = 2 * (y - target)
dw, db = dloss_dy * x, dloss_dy * 1
w, b = w - lr * dw, b - lr * db
print(f"step {step}: loss {loss:.3f}, dw {dw:.2f}, db {db:.2f}")Output
step 0: loss 64.000, dw -32.00, db -16.00 step 1: loss 16.000, dw -16.00, db -8.00 step 2: loss 4.000, dw -8.00, db -4.00
Related terms
More Deep learning core terms
- Backpropagation through time
- Backward pass
- Batch normalisation
- Batch renormalisation
- Batch size
- bfloat16
- Bias term
- Bidirectional RNN
Where you learn it
Backpropagation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.