trendingAI & ML — INTERMEDIATE
Variance, Standard Deviation, and Confidence Intervals for Eval Scores
The Introduction level covered why a single eval score can be misleading in general terms. This level covers the actual formulas: how to compute variance and standard deviation by hand on a real test set, why sampling temperature is a concrete, mechanical source of run-to-run score variance, and how to build a real confidence interval instead of just eyeballing whether a gap between two models looks big enough to matter.
The Quick Answer
Variance is the average of the squared distance between each score and the mean; standard deviation is its square root, which puts it back on the same scale as the original scores. Together they tell you how spread out a set of eval scores actually is, on top of just the average. A confidence interval combines the mean, the standard deviation, and the sample size into a range — commonly the mean plus or minus 1.96 times the standard error (standard deviation divided by the square root of n) for a 95% confidence level — that expresses how much uncertainty is really behind a reported score. And because standard error shrinks with the square root of sample size, not sample size itself, getting a meaningfully tighter interval takes a disproportionately bigger test set.
Computing Variance and Standard Deviation by Hand
Given a set of per-example scores, variance is computed in three steps: find the mean, subtract the mean from every individual score and square each difference (squaring makes every difference positive, so scores above and below the mean don’t cancel out), then average those squared differences. Standard deviation is just the square root of that variance, undoing the squaring so the result is back in the same units as the original scores — which is why standard deviation, not variance, is usually the number reported alongside a mean.
As a concrete worked example: scores of [0.9, 0.7, 1.0, 0.4, 0.8] have a mean of 0.76. The squared differences from that mean are [0.0196, 0.0036, 0.0576, 0.1296, 0.0016], which average to a variance of about 0.0424 — so the standard deviation is roughly √0.0424 ≈ 0.206. That 0.206 says: individual scores typically sit around 0.2 away from the mean of 0.76, which is a fairly wide spread for a 0–1 scale.
Try it yourself — toggle examples in and out and watch both numbers update together:
Sampling Temperature as a Mechanical Source of Variance
At each generation step, a language model produces a probability distribution over its entire vocabulary for what token comes next. Temperature reshapes that distribution before a token gets sampled from it: a temperature of 0 collapses the choice down to always picking the single highest- probability token (deterministic, no randomness at all), while a temperature above 0 flattens the distribution, giving lower-probability tokens a real, non-zero chance of getting picked. This is exactly why re-running the same prompt at temperature 0.7 can produce a genuinely different response each time, while temperature 0 produces the same response every time (barring other sources of nondeterminism like hardware-level floating-point variation). If a response can come out differently, its downstream eval score can come out differently too — this is variance you introduced by the temperature setting itself, not variance in how good the underlying model actually is.
Adjust the temperature and re-run the simulated eval below to see this directly:
Standard Error and Building a Real Confidence Interval
Standard deviation describes spread across individual examples. Standard error is a related but different quantity: it describes how much the mean itself would likely vary if you re-sampled a new test set of the same size and re-measured. Standard error is computed as the standard deviation divided by the square root of the sample size (SE = SD / √n). From there, a 95% confidence interval is commonly built as the mean plus or minus 1.96 times that standard error — 1.96 being the standard z-score that captures the middle 95% of a normal distribution.
Enter your own mean, standard deviation, and sample size below and watch the interval recompute in real time:
Why the Square Root Matters
Because standard error divides by √n rather than n, the relationship between sample size and confidence is not linear. Going from n = 100 to n = 400 (four times as many examples) only halves the standard error, since √4 = 2. Going from n = 100 to n = 10,000 (a hundred times as many examples) only shrinks the standard error by a factor of 10, since √100 = 10. This is the mathematical reason a small eval bump in sample size — say, 100 to 150 examples — barely tightens a confidence interval at all, while getting a meaningfully tighter interval requires an order-of-magnitude larger test set.
Revisit the sample-size explorer with this in mind — notice how much sample size has to grow before the observed gap stabilizes:
Reading a “Model A Beats Model B” Claim Correctly
Put variance, standard error, and sample size together and you get the actual question worth asking about any model-comparison claim: is the observed gap between two models large relative to the uncertainty around each of their scores, or is it small enough to plausibly be noise? A gap of 2 percentage points backed by tight confidence intervals on both models is a much stronger claim than the identical 2-point gap backed by wide, overlapping intervals.
This is the same overlap idea from the Introduction level, now with the actual numbers behind it:
For what these scores are actually measuring — automated metrics, human evaluation, and LLM-as-judge — see Evaluating LLM Outputs: Intermediate. This page’s job is reading whatever score comes out of those methods with appropriate statistical care, not re-explaining what the methods themselves do.
Fun Fact
The 1.96 multiplier used for a 95% confidence interval isn’t an arbitrary round number — it’s the specific z-score at which exactly 95% of a standard normal distribution’s area falls between −1.96 and +1.96 standard deviations from the mean. Different confidence levels use different multipliers: 90% confidence uses roughly 1.645, and 99% confidence uses roughly 2.576 — a stricter confidence level always widens the interval, since claiming more certainty requires casting a wider net.
Test Yourself
How is variance computed for a set of eval scores?
What is standard deviation, in relation to variance?
Why can evaluating a model at a higher sampling temperature increase the variance of its eval scores across repeated runs?
A 95% confidence interval around a mean eval score of 0.80 is computed as [0.75, 0.85]. What does this interval actually communicate?
Why does quadrupling your test set size only cut the standard error in half, rather than to a quarter?
Brainstorm
No right answers here — just questions worth sitting with for a moment before you expand each one to see one way of thinking about it.
Two eval runs report the same mean score but very different standard deviations. What would you want to investigate differently for each one?
If you could only afford to evaluate on 50 examples instead of 500, what would you do differently to make that smaller evaluation more trustworthy?
Why might a team choose to report a confidence interval alongside an eval score instead of just the score itself, even though it makes the result look "less impressive"?
If a model's scores had zero variance across a test set (every single example scored exactly the same), what would that tell you, and would it necessarily be a good sign?
When comparing two models, would you rather have a small confidence interval around a small average gap, or a wide confidence interval around a large average gap? Walk through your reasoning.
Ready for a real paired significance test worked in code, and how to decide whether an observed gap between two models is actually statistically meaningful? Continue to the Advanced level →