Paired Significance Testing for Model Comparisons

Why Pairing Changes the Comparison

Suppose Model A and Model B are both run on the same 100-example test set. An unpairedcomparison would treat Model A’s 100 scores and Model B’s 100 scores as two separate, unrelated samples, and ask whether their means differ by more than you’d expect from two independent samples’ natural noise. But that throws away real information: if example #47 happens to be unusually hard, both models will likely score lower on it — that shared difficulty is noise for the comparison, not signal. A pairedcomparison looks at each example’s difference (Model A’s score minus Model B’s score on that same example) directly, which cancels out exactly this kind of shared per-example variation and leaves a cleaner signal of which model is actually better.

The t-statistic for this paired difference is computed the same way any t-statistic is: the mean of the differences, divided by the standard error of the differences (their standard deviation divided by the square root of the number of paired examples). Try different scenarios below and watch how sensitive the resulting t-statistic is to both the size of the per-example gaps and how consistent those gaps are:

The Multiple Comparisons Problem

A standard significance threshold is typically set so that, if there really were no difference between two models, you’d still expect to see a “significant” result purely by chance about 5% of the time on any single test. That 5% figure only holds for one test in isolation. Run 20 independent comparisons — 20 different metrics, 20 different baselines, 20 slices of a test set — and, with no real difference anywhere, you should expect roughly one of those 20 to look “significant” by chance alone. This is the multiple comparisons problem, and it is a real, well-documented statistical hazard, not a theoretical nitpick — reporting the one metric out of twenty that happened to favor your model, without disclosing the other nineteen, is exactly the failure mode this problem describes.

The standard, widely taught correction for this is the Bonferroni correction: if you’re running m independent comparisons and want an overall 5% false-positive rate across all of them combined, require each individual comparison to clear a stricter threshold of 0.05 / m, rather than 0.05 on its own. It’s a conservative fix — it can make genuinely real, smaller differences harder to detect — but it directly addresses the problem of accumulating false positives across many simultaneous tests.

Source: Bonferroni correction — standard statistical reference

Combining Uncertainty Across Repeated Runs

The Intermediate level showed that sampling temperature above 0 makes repeated runs of the same eval disagree. In practice, this means a careful model comparison should not rely on a single run of each model — it should run each model multiple times and treat the run-to-run variance as an additional source of uncertainty, on top of the per-example variance within any one run. A confidence interval built from only one run understates the model’s true uncertainty, because it’s blind to how much that one run could have differed from another run of the identical model.

Revisit the repeated-run simulator with this framing: each run below is one sample of many possible runs, and a rigorous comparison would average across several of them, not trust any single one:

And the same confidence-interval math from the Intermediate level still applies here — just built from means and standard deviations that reflect multiple runs, not one:

Putting It Together: Reading a Real Comparison Claim

A statistically defensible “Model A beats Model B” claim needs several things in place at once: paired scores on the identical test set (not two separately sampled sets), a sample size large enough that standard error is small relative to the observed gap, a t-statistic (or equivalent) computed from that paired data rather than eyeballed from two separate means, and — if the claim comes from one comparison among many that were run — an acknowledgment of the multiple comparisons problem rather than silently reporting only the favorable result. None of this changes what the eval methods themselves measure, which is covered in The Real Biases in LLM-as-Judge Evaluation — it changes how much weight a resulting number deserves once you have it.

One more look at the overlap picture from earlier levels, now with the full statistical vocabulary behind it — a gap is only as strong as the significance test, sample size, and paired structure that produced it:

Fun Fact

The t-distribution used in significance testing was developed by William Sealy Gosset, a chemist at the Guinness brewery, who published it in 1908 under the pseudonym “Student” — company policy at the time restricted employees from publishing under their own names — which is why it’s still called “Student’s t-distribution” today.

Source: Student (W. S. Gosset), “The Probable Error of a Mean,” Biometrika, 1908

Test Yourself

Brainstorm

No right answers here — just questions worth sitting with for a moment before you expand each one to see one way of thinking about it.