Gradient Descent at Scale: Batches, Momentum, and Adam

Batch, Stochastic, and Mini-Batch Gradient Descent

Every gradient descent example so far computed the gradient from one data point. In practice, there are three different choices for how much data feeds into each gradient calculation before taking a step:

“SGD” in most modern deep learning contexts, including in the Adam paper itself, actually refers to mini-batch gradient descent in this practical sense, not the strict single-example version — the underlying update rule from the Intermediate level is identical either way; only how many examples get averaged into one gradient calculation changes.

Whichever batch size feeds it, the step itself is still exactly this same downhill move — worth re-grounding in the simplest case before adding momentum and Adam on top:

The Learning Rate Is Still the Lever

Everything below builds on top of the update rule, but none of it removes the learning rate’s basic overshoot-vs-crawl trade-off from the Intermediate level — it just gives training more sophisticated ways to manage that trade-off automatically. Worth revisiting the three trajectories side by side before going further:

Momentum: Using the Path You’ve Already Taken

Plain gradient descent reacts only to the gradient at the current step, with no memory of where it came from. Momentum changes that: instead of stepping by the raw current gradient, it steps by a running average that blends the current gradient with the accumulated direction of recent steps.

This matters most in a real, high-dimensional loss landscape shaped like a narrow ravine — steep in one direction, shallow in another. Plain gradient descent tends to bounce back and forth across the steep direction while making frustratingly slow progress along the shallow one. Momentum’s running average tends to cancel out that bouncing (since it keeps flipping sign and averages toward zero) while reinforcing the direction that’s been consistent — letting the update move more directly toward the minimum instead of zig-zagging.

Adam: A Learning Rate Per Weight

Adam (Adaptive Moment Estimation), introduced by Kingma and Ba, combines momentum’s running average of the gradient with a second running average — of the squared gradient — and uses both together to give each individual weight its own adapted effective learning rate, rather than every weight sharing one fixed global value.

The paper describes the method as computing “individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients,” and reports it works well in practice “for problems that are large in terms of data and/or parameters” with “little memory requirements.” In plain terms: weights with consistently large, noisy gradients get automatically smaller effective steps, and weights with small or infrequent gradients get comparatively larger ones — exactly the kind of per-weight nuance that becomes valuable once a model has millions or billions of weights with very different roles, unlike the small toy examples used earlier in this series.

Source: Kingma & Ba, “Adam: A Method for Stochastic Optimization,” arXiv:1412.6980

Revisit the multi-weight update from the Intermediate level with this in mind — Adam doesn’t change what a gradient means or what the update rule fundamentally does, it changes how the step size for each individual weight’s update gets chosen:

The Same Arithmetic Underneath All of This

Whatever optimizer sits on top — plain gradient descent, momentum, Adam — the core arithmetic each of them ultimately computes is the same weight-and-gradient relationship worked out here. Adam and momentum only change how the learning-rate-and-gradient product gets shaped before it’s subtracted:

Local Minima and Learning Rate Schedules

The local-vs-global-minima problem from the Intermediate level doesn’t go away with a better optimizer — but momentum and adaptive methods do help in practice, since accumulated momentum can carry a weight update through a shallow dip that would have stopped plain gradient descent cold, the same way a ball rolling with real momentum can coast through a small dip instead of settling in it.

This is also part of why many real training runs use a learning rate schedulerather than one fixed value for the whole run: starting with a larger learning rate makes fast, broad progress across the landscape (and gives the optimizer a better chance of not settling permanently in an early shallow dip), while gradually decreasing it later lets the model settle precisely near a minimum without the overshoot risk a large learning rate would carry once it’s already close.

Where This Connects Across the Site

Every fine-tuning method covered elsewhere on this site is this same machinery, applied to a specific loss and a specific set of trainable weights: full fine-tuning updates all of a model’s weights with this loop; LoRA restricts which weights are trainable but runs the identical update rule on them; and the reward-model and policy training inside RLHF are each their own gradient descent loop, just with a loss function built from human preference data instead of a fixed label. What efficiently computes the gradient for every weight in a deep, many-layered network — rather than by hand, one weight at a time, as done throughout this topic — is backpropagation, the natural next stop after this one.

Fun Fact

Despite Adam being introduced in 2015 as one optimizer among many being actively studied, its original paper has gone on to become one of the most-cited papers in all of machine learning — a reflection of just how much of modern deep learning training, across completely different model architectures and tasks, still runs on some close variant of the method it introduced.

Source: Kingma & Ba, arXiv:1412.6980

Test Yourself

Brainstorm

No right or wrong answers here — just questions worth pausing on before you expand the model answer.