Next-Token Prediction Explained: Tokenization, Softmax & Sampling

From Text to Numbers

Before any of that happens, raw text has to become numbers. Modern LLMs use a subword tokenizer — commonly Byte-Pair Encoding (BPE) — that starts from individual characters/bytes and iteratively merges the most frequent adjacent pairs until it hits a fixed vocabulary size. That’s why “strawberry” doesn’t map to one token: it splits into whatever subword pieces the tokenizer happened to learn during training.

Each resulting token ID is just an index into a lookup table. That table maps every token ID to a dense vector — an embedding, typically a few thousand dimensions — and it’s that vector, not the raw ID, that actually flows into the model’s layers.

The Actual Prediction Step

After the input passes through the model’s transformer layers, the final layer produces one logit per vocabulary entry — a raw, unnormalized score for every possible next token. Softmax turns that into a real probability distribution:

That guarantees every probability sits between 0 and 1 and the whole distribution sums to exactly 1 — well-formed odds over the entire vocabulary, recomputed from scratch at every single token position.

Sampling: Why “Most Likely” Isn’t Always Picked

Softmax gives you a distribution, not a decision. What happens next depends entirely on the sampling strategy:

• Greedy decodingalways takes the single highest-probability token — deterministic, but prone to repetitive, “safe” output.
• Temperature sampling divides every logit by a temperature value before softmax runs. Lower temperature sharpens the distribution toward the top candidate; higher temperature flattens it, giving lower-probability tokens a real shot.
• Top-k sampling restricts the candidate pool to the k highest-probability tokens before sampling, discarding the long tail entirely.
• Top-p (nucleus) sampling keeps the smallest set of tokens whose cumulative probability exceeds p, so the pool size adapts to how confident the model already is.

“

What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.

Try it below: same 5 candidate tokens, real softmax math, live temperature control.

The Context Window, Technically

The context window is the fixed maximum number of tokens — input and generated output combined — the model can attend to in a single forward pass. Every generation step re-processes the entire sequence up to that point, so wall-clock cost per token grows with sequence length. Once the window fills up, the oldest tokens fall out of view entirely — the model has no memory of anything before that point unless a chat client explicitly reinserts it from an earlier turn.

Why Typos Still Resolve Correctly

Earlier you saw the same garbled letters — “dg” — resolve to “dog” in one sentence and “dig” in another. Technically, this works because the surrounding tokens’ embeddings, once passed through several transformer layers of self-attention, shift the model’s internal representation of the ambiguous token toward whichever region of embedding space its neighbors point to. Nearby words directly bias the logits the final layer produces for that position, before softmax ever runs — and a garbled token still lands somewhere identifiable in that space, because the tokenizer usually preserves recognizable subword fragments even from misspelled input.

The Generation Loop in Pseudocode

tokens = tokenize(prompt)
while not done:
    logits = model(tokens)            # one score per vocab entry
    probs  = softmax(logits / temperature)
    next_token = sample(probs)        # greedy, top-k, or top-p
    tokens.append(next_token)
    done = next_token == EOS or len(tokens) >= max_tokens
return detokenize(tokens)

Fun Fact

GPT-4o’s tokenizer, o200k_base, has a vocabulary of just over 200,000 entries. That means every single softmax computation during generation produces a probability distribution across all 200,000+ of those possible next tokens — recalculated from scratch at every position in the output.

Test Yourself

Ready for the internals — self-attention, KV caching, speculative decoding, and perplexity? Continue to the Advanced level →