What is backpropagation in simple terms?

Backpropagation is the process a neural network uses to figure out how much each of its internal weights contributed to a mistake, by working backward from the error at the output through every layer. It answers the question "which weights should change, and by how much" so the network can get a little better on the next try.

Is backpropagation the same thing as gradient descent?

No — they are two different steps that work together. Gradient descent is the rule for updating a weight once you know its gradient (move it a little in the opposite direction). Backpropagation is how that gradient actually gets computed in a network with many layers, by applying the chain rule backward from the output.

AI & ML — INTRODUCTION

Backpropagation: How a Network Figures Out What Went Wrong

A network makes a prediction, the prediction is wrong, and somehow every single one of its thousands (or billions) of internal weights ends up nudged in a slightly better direction. Backpropagation is the process that makes that possible — and it is exactly what the topic on Gradient Descent assumes is already available before it can do its job.

The Quick Answer

Backpropagation is how a neural network works out which of its weights caused a mistake, and by how much — by starting at the output error and working backward, one layer at a time, back toward the input. It doesn’t guess. Each weight gets an exact share of the blame for the final error, computed using a rule from calculus called the chain rule. That per-weight blame value is called the gradient, and it’s the exact thing gradient descent needs before it can move any weight at all.

Working Backward from a Mistake

Imagine three people each did one step of baking a cake, and the cake came out wrong. Figuring out who to blame — and how much — means starting at the finished (bad) cake and working backward through the steps: did the last step ruin an otherwise fine cake, or was the mistake baked in from an earlier step? A network faces the exact same problem after a wrong prediction, just with weights standing in for the workers.

Two Passes Through the Network

Every time a network processes one training example, it actually does two separate passes through its layers. The forward pass is the familiar one: input goes in, flows through each layer in order, and a prediction comes out. The backward passis backpropagation itself: the error at the output flows back through the same layers in reverse, computing each layer’s share of the blame along the way.

Try both directions below and notice which layer gets its gradient first during the backward pass — it’s the layer closest to the output, not the input.

Why Not Just Test Each Weight One at a Time?

There is a simpler-sounding way to find out how much a weight matters: nudge it slightly, re-run the whole network, and see how much the error changed. Do that for every single weight, one at a time. It would work — but a realistic network can have millions or billions of weights, and re-running the entire network once per weight is enormously slow.

Backpropagation gets the exact same information — the gradient for every weight — from just one forward pass and one backward pass, total, no matter how many weights the network has. That efficiency is the entire reason training deep networks with many layers is practical at all.

The Chain Rule, in Plain Terms

The tool that makes this efficient backward calculation possible is a rule from calculus called the chain rule. In plain terms: if changing A changes B, and changing B changes C, then the chain rule lets you find out exactly how much changing A ends up changing C — by multiplying together how much A affects B and how much B affects C. A network is a long chain of exactly this kind of dependency, layer after layer, and the chain rule is what lets the effect of an early weight on the final error be computed without ever re-running the network from scratch.

Below is a tiny, real, numeric example — not a full network, just enough steps to see the chain rule actually multiply through:

Fun Fact

The core idea behind backpropagation — using the chain rule to compute gradients efficiently through a layered computation — was described for neural networks in a short 1986 Naturepaper by Rumelhart, Hinton, and Williams. It wasn’t the very first appearance of the underlying math, but it’s the paper most widely credited with showing the machine learning community that it made training multi-layer networks practical.

Source: Rumelhart, Hinton & Williams, “Learning representations by back-propagating errors,” Nature 323, 533–536 (1986)

Test Yourself

What question is backpropagation actually answering?

Which direction does backpropagation move through the network?

How is backpropagation different from gradient descent?

Brainstorm

No right answers here — these are worth pausing on before you move to the next level.

If a network has one wrong output, why would it make sense to check the last layer’s weights before the first layer’s weights?

The baking analogy above assigned "blame" as percentages. What would it mean for a real network weight to get 0% of the blame for a mistake?

Why might a network need to see the same mistake repeated across many different examples before a weight changes meaningfully?

The forward pass computes a prediction; the backward pass computes blame. Could a network learn anything using only the forward pass, with no backward pass at all?

Why do you think this process is called "backpropagation" rather than something like "error correction"?

Ready to see the real chain-rule math worked through an actual multi-layer network, plus the vanishing gradient problem? Continue to the Intermediate level →