How Do LLMs Work? Token-by-Token Prediction Explained

What the Experts Say

“

LLMs work by receiving an input or prompt, calculating what is most likely to come next, and then producing an output or completion.

How It Actually Works

• 1

  "LLM" is just the name. Large language model — a program trained on enormous amounts of text to guess what comes next.

• 2

  One token at a time. It never generates a full sentence in one shot; it produces one token, then feeds that token back in as new input.

• 3

  A token isn’t a whole word. Long or unusual words often get split into smaller pieces before the model ever sees them.

• 4

  Ranked, not random. Each new token is picked from a ranked list of candidates — almost always the one the model considers most likely.

• 5

  Autocomplete, scaled up. Same idea as your phone’s keyboard, just trained on a vastly larger slice of text and far better at picking the next word.

• 6

  Confident ≠ correct. Because the model is predicting patterns, not looking up facts, it can sound completely confident while still being wrong.

Even a Typo Doesn’t Fool It

Type “The cat sta on teh mat” into a chatbot and it still replies as if you spelled everything right — no error, no confusion. “Sta” and “teh” are garbled, but the ranked-candidate trick from above still saves it: the surrounding words — “The cat ___ on ___ mat” — narrow the next-token race so hard that “sat” and “the” win anyway, typo or not. Training data is full of real human typing, typos included, so the model has effectively seen this exact kind of mess before.

Scientists who studied how these programs work found something neat: the AI has specialized inner parts that act almost like a built-in spell-checker. Easy typos get caught fast using just the word or two right next to them. Trickier typos need more help, so the AI looks at the whole sentence for extra clues before deciding what you meant. A 2025 study confirmed this typo-fixing happens automatically — nobody had to teach the AI a specific rule for it.

Source: Tsuji et al., “Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errors,” arXiv:2502.19669 (2025)

Here’s the part that surprises people: the exact same typo can get fixed two completely different ways, depending on what’s written around it. Try it below.

Watch It Happen, Word by Word

Here’s what that looks like in practice. Take the sentence “The cat sat on the mat.” The model never sees all six words as a finished thought — it starts knowing only “The,” and from there it has to rank thousands of possible next words and pick the one it thinks is most likely. Say “cat” wins. Now it looks at “The cat” and asks the exact same question again: what comes next? Do that six times in a row and the full sentence exists — not planned in advance, just assembled one confident guess at a time. The widget below lets you play that same guessing game yourself, one word at a time.

👉 Try it yourself — click a different word at any point above and watch the sentence branch into a completely different ending.

Fun Fact

Ask an LLM to spell “strawberry,” and plenty of them used to answer “2” instead of 3. GPT-4’s tokenizer splits the word into pieces like “st,” “raw,” and “berry” — the model never actually sees individual letters, only these token chunks, which makes counting letters one of the strangest things for it to get wrong.

Test Yourself

Ready for the engineering details — tokenization internals, softmax, and sampling strategies you can tune from an API call? Continue to the Intermediate level →