What is reverse-mode automatic differentiation and how does it relate to backpropagation?

Reverse-mode automatic differentiation is the general algorithm that computes exact gradients of a scalar output with respect to every input in a computational graph, by traversing the graph backward once and applying the chain rule at each node. Backpropagation, as used in neural networks, is reverse-mode automatic differentiation applied specifically to the computational graph defined by a network's layers and loss function.

Why is reverse-mode automatic differentiation preferred over numerical differentiation for training neural networks?

Numerical differentiation approximates each weight's derivative by perturbing it slightly and re-running the forward pass, which costs one full forward pass per weight and only produces an approximation. Reverse-mode automatic differentiation computes exact derivatives for every weight in a network from a single forward pass plus a single backward pass, regardless of how many weights the network has, making it dramatically cheaper at the scale modern networks operate at.

← Back to the Intermediate level

AI & ML — ADVANCED

Backpropagation Internals: Computational Graphs and Reverse-Mode Autodiff

Q: What are real documented fixes for the vanishing gradient problem?

Residual (skip) connections, introduced in the ResNet architecture, give the gradient an additional path back to earlier layers that does not require passing through every intervening layer's local derivative, directly mitigating vanishing gradients in very deep networks. Activation functions with a wider, less saturating derivative range, such as ReLU, are also widely used specifically because their derivative does not shrink toward zero the way sigmoid's does across most of its input range.

The Intermediate level worked through the chain rule layer by layer. This level goes one level deeper: backpropagation as it is actually implemented — as reverse-mode automatic differentiation over a computational graph — plus why that framing beats the naive alternative, and real, documented fixes for the vanishing gradient problem. As before, this page is strictly about computing the gradient; what to do with it once you have it is covered in Gradient Descent & Loss Functions.

The Quick Answer

Backpropagation is reverse-mode automatic differentiation applied to the computational graph a neural network defines. Every operation in that graph — every addition, multiplication, and activation function — is a node that knows only one thing: its own local derivative with respect to its own inputs. A single backward traversal of the graph, applying the chain rule at each node in turn, produces the exact gradient of the loss with respect to every parameter in the network, from one forward pass and one backward pass, regardless of how many parameters exist.

The Computational Graph View

Frameworks like PyTorch build a computational graph as the forward pass executes: every tensor operation becomes a node, and every node records both its inputs and its own local derivative rule. This is a strictly more granular view than “layers” — a single dense layer is really several graph nodes (a matrix multiply, an addition for the bias, an activation function), each with its own simple local rule.

Try a few individual node types below and see exactly how each one routes an incoming gradient back to its own inputs — this is the literal operation real autodiff engines perform, once per node, during the backward pass:

Source: PyTorch documentation, “Autograd mechanics”

Reverse-Mode Autodiff vs. Numerical Differentiation

There is a genuinely different way to estimate a derivative: numerical differentiation, which perturbs a weight by a tiny amount, re-runs the forward pass, and measures how much the loss changed. It is simple to implement and useful as a correctness check, but it only approximates the true derivative (subject to floating-point precision limits), and it costs roughly one full forward pass per weight being checked.

Reverse-mode automatic differentiation gets the exact gradient for every parameter in the graph from a single backward traversal, reusing intermediate values computed once during the forward pass rather than recomputing them per parameter. This is precisely why frameworks use it for training and reserve numerical differentiation for spot-checking gradient correctness on small test cases, not for training at scale.

The Forward/Backward Split, Formalized

Formally: the forward pass computes and caches every intermediate node’s output, topologically ordered from inputs to the final scalar loss. The backward pass then visits those same nodes in exactly reverse topological order, at each one multiplying the gradient flowing in from “downstream” (closer to the loss) by that node’s own local derivative, and passing the result “upstream” to whichever nodes feed into it. By the time the traversal reaches the input weights, each one holds the exact accumulated product the chain rule specifies.

Reprise: A Full Chain-Rule Trace

Here is the same tiny composed network again, worth revisiting at this level as a concrete instance of the general algorithm above: three graph nodes, each contributing exactly one local derivative to the final product.

Vanishing Gradients: The Problem and Real Fixes

As covered at the Intermediate level, multiplying many local derivatives smaller than 1 across a deep graph shrinks the gradient toward zero by the time it reaches early layers. Adjust the depth and derivative size again below to see the exact arithmetic:

Two architectural fixes are directly documented as responses to this problem. First, residual (skip) connections, introduced in the ResNet architecture, add an identity path around a block of layers — since addition’s local derivative is exactly 1, the gradient gets an additional route back to earlier layers that isn’t shrunk by that block’s other local derivatives. Second, ReLU-family activation functionsare widely used specifically because their derivative doesn’t saturate toward zero across most of the positive input range the way sigmoid’s does, keeping local derivatives closer to 1 through more of the network.

Source: He, Zhang, Ren & Sun, “Deep Residual Learning for Image Recognition,” arXiv:1512.03385

Fun Fact

The ResNet paper’s skip connections were originally motivated by an odd, empirically observed problem: adding more layers to an already-deep plain network could make training accuracy worse, not just test accuracy — a symptom the paper explicitly separates from ordinary overfitting, since it showed up on the training set itself, not only on held-out data.

Source: He, Zhang, Ren & Sun, arXiv:1512.03385

Test Yourself

What is backpropagation, precisely, in terms of automatic differentiation?

Why does reverse-mode automatic differentiation beat numerical differentiation for training networks?

How do residual (skip) connections help with the vanishing gradient problem, as introduced in the ResNet architecture?

At the level of an individual computational-graph node (a "gate"), what does backpropagation require that node to know?

Brainstorm

Genuinely open questions worth sitting with — expand each one for one way to think about it, not the only way.

Reverse-mode autodiff computes gradients of one scalar output with respect to many inputs efficiently. If you instead needed the gradient of many outputs with respect to one input, which mode of automatic differentiation (forward or reverse) do you think would be more efficient, and why?

Residual connections give the gradient a path that skips over some layers entirely. Why might that specifically help gradients reach early layers, compared to just using a better activation function alone?

A computational graph node caches its forward-pass inputs for use during the backward pass. What do you think happens to memory usage as a network gets both deeper and wider, and why might that matter for training very large models?

If a single node in the computational graph has a bug in its local derivative rule, what do you expect the practical symptom to look like during training, versus a bug in the forward computation itself?

Gradient descent (the previous topic in this series) assumes a correct gradient is available before it can take a step. What do you think would happen to a training run if backpropagation had a subtle, consistent bias in how it computed one particular layer's gradient?