trendingAI & ML — INTERMEDIATE
Backpropagation and the Chain Rule, Worked Through a Real Network
This topic picks up right after Gradient Descent & Loss Functions, which covers what to do once you have a gradient — move each weight a small step opposite it. This page covers the part gradient descent assumes is already solved: how a network with many layers actually computes that gradient for every single weight, efficiently, using the chain rule.
The Quick Answer
Backpropagation computes the gradient of the loss with respect to every weight in a network by applying the chain rule backward, one layer at a time, starting from the output error. Each layer only needs to know its own localderivative — how much its own output changes with respect to its own input. Multiplying local derivatives together along the path from a weight to the loss gives that weight’s exact gradient, and a single backward pass computes this for every weight in the network at once, by reusing shared intermediate values instead of recomputing them per weight.
Forward Pass, Then Backward Pass
Training on one example always involves two passes through the network. The forward pass computes a prediction, layer by layer, from input to output — and along the way, it caches the intermediate values each layer produced (you’ll see exactly why that caching matters below). The backward pass then walks those same layers in reverse, starting from how wrong the final prediction was, computing each layer’s gradient before handing a signal to the layer before it.
The Chain Rule, Worked With Real Numbers
The chain rule says: if a change to w1 changes h, which changes y, which changes the loss, then the total effect of w1 on the loss is the product of each of those individual (local) effects. Below is a genuinely tiny 3-step network — small enough to compute by hand — where you can adjust the weights and watch every local derivative, and the final chain-rule product, update in real time.
Nothing in that calculation required re-running the network from the very first step again for w1— each local derivative was computed once and reused. Scale this same idea up to a network with millions of weights across many layers, and the pattern is identical: one backward pass, reusing local derivatives layer by layer, produces every weight’s gradient.
Why This Matters: Making Deep Learning Computationally Tractable
The alternative to backpropagation isn’t some other clever algorithm — it’s brute force: nudge one weight, re-run the entire forward pass, measure how the loss changed, then repeat for every other weight, individually. That approach gets exact gradients too, but at a cost that scales with the number of weights, which for any modern network is not a small number.
Backpropagation reduces that cost to one forward pass plus one backward pass, regardless of how many weights the network has, because the chain rule lets local derivatives be computed once and shared across every weight that depends on them. This is not a minor speed-up — it is the specific property that makes training networks with millions or billions of parameters computationally realistic at all.
Every Operation Is a Small, Local Rule
Zoom in further than “layers,” and a network is really built from individual operations — additions, multiplications, activation functions — each of which only needs to know one small, local rule to participate in backpropagation: given the gradient flowing in from the step after it, how should that gradient be passed on to its own inputs? Try a few different operations below and see how differently each one routes the incoming gradient.
The Vanishing Gradient Problem
The chain rule’s multiplication has a real downside in very deep networks. If the local derivative at each layer tends to be smaller than 1 — which is common with activation functions that squash their output into a narrow range — then multiplying many of these small numbers together across many layers shrinks the product toward zero. By the time that gradient reaches the earliest layers of a deep network, it can be so small that those layers barely update at all during training, even though they may still matter for the network’s output.
Adjust the depth and the size of the local derivative below and watch the running product shrink — this is the exact arithmetic behind the vanishing gradient problem, not a simplification of it.
This phenomenon is well documented in the deep learning literature as a real obstacle to training very deep networks with earlier activation functions, and it directly motivated later architectural fixes (residual connections, better-behaved activation functions, normalization layers) that are outside the scope of this page but worth knowing exist.
Fun Fact
A single multiply operation’s local gradient with respect to one input is simply the otherinput’s forward value. That is the entire reason real backpropagation implementations cache every layer’s forward-pass outputs in memory during training — the backward pass genuinely needs those exact numbers again, not just the computation graph’s shape.
Test Yourself
In a network with layers A → B → C → loss, what does the chain rule let you compute the gradient of the loss with respect to a weight in layer A as?
Why is backpropagation dramatically more efficient than nudging each weight individually and re-running the network?
What causes the vanishing gradient problem in a deep network?
What is the actual difference between backpropagation and gradient descent?
Brainstorm
No single right answer for these — think them through before checking the model answer.
If a weight sits right next to the loss (the very last layer), do you think its gradient computation involves fewer or more multiplications in the chain than a weight in the first layer? Why?
The vanishing gradient demo showed the running product shrinking as depth increases. What do you think would happen to the running product if the local derivative were consistently greater than 1 instead of less than 1?
Why does backpropagation need to remember (cache) the forward pass’s intermediate values, rather than just computing the backward pass from scratch?
Two networks have the same number of layers, but one uses an activation function whose derivative is often close to 1, and the other uses one whose derivative is often close to 0.2. Which one do you expect to train more easily at greater depth, and why?
Gradient descent (covered in the previous topic) needs a gradient for every weight before it can update anything. What would training even look like if you only had an approximate, noisy version of that gradient instead of the exact one backpropagation computes?
Ready for the full multi-layer derivation in matrix form, how automatic differentiation engines implement this in real code, and documented fixes for vanishing gradients? Continue to the Advanced level →