Gradient Descent & Loss Functions: How a Model Actually Learns

What a Loss Function Actually Is

A model’s job is to make a prediction — the next word, a price, a label. A loss function is simply the rule that scores how wrong that prediction was, compared to the correct answer, as one number. Guess exactly right and the loss is zero. Guess further off and the loss is bigger. There are different exact formulas for different kinds of tasks, but they all share this same shape: smaller number means better prediction.

A common, simple version squares the difference between the guess and the correct answer. Try it below — slide the guess around and watch the loss number react:

The Loss Landscape: A Bowl You’re Trying to Roll Into

Now imagine plotting the loss for every possible value a weight could take, instead of just one guess. That picture — loss on one axis, the weight’s value on the other — is called the loss landscape. For a simple case it looks like a bowl: high loss at the edges, a low point somewhere in the middle. Training a model is the process of finding your way toward that low point, without ever seeing the whole bowl at once — only feeling the slope right where you’re currently standing.

That’s exactly what gradient descent does: at the current weight, it checks which direction is downhill (the gradient) and takes a step that way. Try it yourself below — press “Step” repeatedly and watch the marker roll toward the bottom of the bowl, with the loss number dropping with each step:

One Update, Done by Hand

It helps to see the actual arithmetic behind one of those steps, on a model simple enough to compute by hand. Picture the simplest possible model: prediction = weight × input. Pick an example below and walk through exactly how the weight changes after one step:

The rule doing all the work there is always the same one: new weight = old weight − learning rate × gradient. Everything about gradient descent, at every scale — from this one-weight toy example up to a model with billions of weights — comes down to that same line, applied over and over.

The Learning Rate: How Big a Step to Take

The learning rate in that formula controls how big each step is. Get it wrong in either direction and training suffers in a very concrete way:

• Too high

  Each step is so large it overshoots past the bottom of the bowl entirely, landing somewhere worse. Training can bounce around or even get worse over time instead of better.

• Too low

  Each step barely moves the weight at all. The model eventually gets to a good answer, but training takes far longer — and far more compute — than it needs to.

• Just right

  Steps are sized to make real progress each time without overshooting past the goal — the fastest realistic path to a low loss.

This same idea — that the exact same starting point can converge cleanly, crawl slowly, or diverge entirely, depending only on step size — gets a hands-on, side-by-side comparison in the Intermediate level of this topic.

Not Every Bowl Has Just One Bottom

The bowl picture above is the simplest case. Real loss landscapes can have more than one dip — a shallow one and a deeper one. Because gradient descent only ever feels the local slope, it can settle into a shallow dip and stop, even though a better, deeper dip exists somewhere else entirely. Try starting from two different points below and see where each one ends up:

Where This Actually Shows Up

This isn’t just a training-a-model-from-scratch idea. Every time a model gets fine-tuned on new examples, adapted with a method like LoRA, or shaped by human feedback in RLHF, the underlying mechanism making its weights actually change is exactly the loop covered here: measure the loss, take a gradient-descent step, repeat. What differs across those topics is the data, the loss function, and how many weights get updated — not this core loop.

Fun Fact

Gradient descent as a mathematical method predates modern AI by over a century — the French mathematician Augustin-Louis Cauchy described the core method of iteratively stepping in the direction of steepest descent to solve a system of equations back in 1847, long before anyone was training neural networks with it.

Source: Cauchy, “Méthode générale pour la résolution des systèmes d’équations simultanées,” Comptes Rendus de l’Académie des Sciences, vol. 25, pp. 536–538, 1847

Test Yourself

Brainstorm

No right or wrong answers here — just questions worth pausing on before you expand the model answer.

Ready to see this same loop with real, worked learning-rate trajectories side by side, and a proper look at what “gradient” means with more than one weight? Continue to the Intermediate level →