What does the temperature parameter actually do to the softmax formula?

Temperature divides every logit by the temperature value before softmax runs its usual exponentiate-and-normalize steps. Dividing by a temperature below 1 widens the gaps between logits, making the resulting distribution more peaked (more confident). Dividing by a temperature above 1 shrinks those gaps, flattening the distribution toward uniform.

Why can top-k and top-p sampling select different numbers of candidate tokens for the same distribution?

Top-k always keeps a fixed count of tokens, regardless of how probability is actually distributed among them. Top-p instead keeps however many tokens are needed for their cumulative probability to reach a threshold p, so its selected set size shrinks on confident, sharply peaked distributions and grows on flatter, more uncertain ones.

What is entropy in the context of a language model’s output distribution?

Entropy is a single number, measured in bits, that quantifies how spread out a probability distribution is. It is 0 when one token holds all the probability (fully predictable) and reaches its maximum value of log2(n) only when all n candidate tokens are equally likely — the same underlying quantity that a higher sampling temperature increases.

← Back to the Introduction level

MATHS FOR AI/ML — INTERMEDIATE

The Real Temperature Formula: Softmax Math Behind Sampling

The Introduction level covered what softmax is and the basic greedy-vs-sampling, top-k-vs-top-p choices. This level goes into the actual math underneath those choices: the real temperature formula, why probability ratios between tokens grow exponentially with logit gaps, entropy as a concrete number for “how spread out” a distribution is, and exactly why top-k and top-p don’t always agree.

The Quick Answer

The softmax formula itself is fixed, but every logit going into it can first be divided by a temperature value:

P(x_i) = \frac{e^{z_i / T}}{\sum_{j} e^{z_j / T}}

At T = 1, this is plain softmax. A lower T widens the gaps between logits before exponentiation, sharpening the distribution toward the top candidate. A higher T shrinks those gaps, flattening the distribution toward uniform. The amount of “spread” in a distribution has a real name and a real number attached to it — entropy— and the choice between top-k and top-p sampling comes down to whether you want a fixed-size or an adaptive-size set of surviving candidates.

The Actual Temperature Math

Prompt Engineering already covers what temperature doesat a conceptual level: lower is more focused, higher is more random. Here’s the arithmetic that actually produces that effect. Every logit is divided by T beforethe usual exponentiate-and-normalize steps. Watch the middle column change as you move the slider — that division is the entire mechanism:

Why Small Logit Gaps Create Big Probability Differences

Softmax has a clean mathematical property worth internalizing: the probability ratio between any two tokens depends only on the difference between their logits, and that relationship is exponential:

\frac{P(a)}{P(b)} = e^{\,z_a - z_b}

This means a logit gap of 2 doesn’t make one token “twice as likely” — it makes it roughly e² ≈ 7.4 times as likely. Small differences in raw model output compound into large probability differences very quickly:

Entropy: A Real Number for “Peaked vs. Flat”

“Peaked” and “flat” are useful words, but they’re not precise. Shannon entropy gives the same idea an exact number, in bits:

H(p) = -\sum_{i} p_i \log_2 p_i

Entropy is 0 when a single token holds all the probability, and reaches its maximum, log₂(n), only when all n candidates are exactly equally likely. Higher temperature doesn’t just informally “add randomness” — it measurably raises this exact number:

Where Top-k and Top-p Actually Disagree

Top-k and top-p both restrict the candidate set before sampling, but they use different rules, and those rules can select entirely different numbers of survivors on the exact same distribution. Top-k always keeps a fixed count; top-p keeps a variable count based on cumulative probability. Adjust both independently below and watch each token’s kept/cut status under each method:

Try It on Your Own Logits

Every demo so far has used curated numbers. Type in your own logits below — including equal values or negative ones — and confirm softmax’s guarantee holds every time: outputs between 0 and 1, always summing to exactly 100%.

Fun Fact

The word “softmax” is a portmanteau of “soft” and “maximum” — it’s a smoothed, differentiable stand-in for the “argmax” operation (just picking the single largest value outright). “Soft” is exactly what temperature adjusts: at very low temperature, softmax behaves almost exactly like a hard argmax; at high temperature, it moves further from it.

Test Yourself

What does dividing every logit by a temperature value less than 1 do to the resulting softmax distribution?

Why can top-k and top-p select a different number of surviving tokens on the same probability distribution?

According to the relationship p_a / p_b = exp(z_a − z_b), what happens to the probability ratio between two tokens as the gap between their logits grows?

What does an entropy of 0 bits mean for a model’s output distribution over the next token?

Brainstorm

These don’t have a single correct answer — they’re here to make you pause and think before checking one way to reason about each one.

If you set temperature to a very large number (say, 100) on a set of logits that aren’t identical, what would you expect the resulting distribution to look like, and why?

Could you set temperature to exactly 0? What do you think would happen mathematically, and how do production systems handle this?

Why might a fixed top-k of, say, k=5, work poorly both for a very confident distribution (one token at 95%) and a very uncertain one (20 tokens roughly tied)?

Two different next-token distributions could have the same entropy value but look very different when you plot them. What might that look like, and why does entropy alone not tell the whole story?

If a logit gap of 2 makes one token about 7.4x more likely than another, what gap would you guess produces roughly a 100x difference, and how would you check it?

Ready for why naive softmax can actually overflow on real hardware, the exact loss function that trains a language model at every step, and how nucleus sampling’s set size adapts to the shape of the distribution? Continue to the Advanced level →