Statistics for Model Evaluation: Why One Eval Score Isn’t Enough

The Mean: One Number Standing in for a Whole Test Set

When an AI model is evaluated, it’s almost never graded on just one question — it’s run against a test set, a whole collection of example questions, each scored individually (right or wrong, or a partial score like 0.7 out of 1). The single headline number you see reported — “82% accuracy,” “0.75 average score” — is the mean: add up every example’s score and divide by how many examples there were. It’s a genuinely useful summary, but it is still just a summary — two very different test sets could produce the exact same mean.

Try it yourself below: toggle examples in and out of a small test set and watch the mean recompute live.

Same Average, Very Different Models

Here’s the case that trips people up: two models can post the identical mean score while behaving completely differently underneath. Imagine Model A scores 0.7 on every single example — consistent, unremarkable, predictable. Now imagine Model B scores 1.0 on half the test set and 0.4 on the other half. Both average out to 0.7. But if you cared about worst-case reliability — say, this model is answering medical or legal questions — Model B’s failures are far more concerning than Model A’s, even though their average score is identical. This is exactly why looking only at the mean, and never at how spread out the scores were, can hide the thing you actually cared about.

See this play out with two models whose average gap looks the same but whose underlying spread is very different:

Why Running the Same Eval Twice Can Give a Different Score

Most language models in production don’t generate the single most likely response every time — they use sampling with a temperature setting above 0, which deliberately introduces some randomness into which words get picked, so the same prompt can produce a genuinely different response on different runs. The concept of how models generate text, one token at a time, is covered from the ground up in How LLMs Work. The consequence for evaluation: if you score a model on the same test set twice, and the model samples with any randomness, you can get two different scores from two runs of the exact same model on the exact same questions — nothing about how good the model actually is has changed, only the random draw.

Try running the same simulated eval twice at different temperature settings:

Why Sample Size Changes How Much You Can Trust a Result

A claim like “Model A beats Model B” depends heavily on how many examples that claim was measured on. Test on only 10 examples, and a couple of lucky or unlucky guesses can flip which model looks better, purely by chance. Test on 1,000 examples, and that chance-driven noise mostly averages out, leaving a result that’s much closer to each model’s true, underlying performance. Neither model actually got better or worse by testing more of it — you just got a clearer picture of what was already true.

Pick different sample sizes below and watch how much the observed gap between two models bounces around, even though the true gap between them never changes:

This site’s Evaluating LLM Outputs topic covers what you’re actually measuring — automated metrics, human evaluation, and LLM-as-judge — in depth. This page is about the statistics underneath any of those methods: how many examples you need, and how much a single number can be trusted, no matter which evaluation method produced it.

Fun Fact

Many widely used LLM benchmarks run each model at a fixed, low or zero temperature specifically to reduce run-to-run randomness and make results reproducible — MMLU, for example, is commonly evaluated with greedy (temperature 0) decoding in its original released evaluation code, precisely because comparing models fairly requires controlling for the randomness a higher temperature would otherwise introduce.

Source: Hendrycks et al., MMLU reference implementation, GitHub: hendrycks/test

Test Yourself

Brainstorm

No right answers here — just questions worth sitting with for a moment before you expand each one to see one way of thinking about it.

Ready for the actual math behind standard deviation and confidence intervals, and how to compute them on a real test set? Continue to the Intermediate level →