trendingAI & ML — INTERMEDIATE
LLM Benchmarks Explained: MMLU, HumanEval & Their Limits
“Task fit” from the Introduction level often gets checked with a public benchmark score. That’s useful — but only if you know exactly what the benchmark is actually testing, and where it stops being a reliable signal.
The Quick Answer
Public benchmarks like MMLU (broad knowledge) and HumanEval (code generation) give a standardized, comparable score across models — genuinely useful when your task resembles what the benchmark tests. Ollama’s own library page is a practical, hands-on comparison tool in its own right: model tags show exact parameter sizes, and pull counts are a real (if imperfect) signal of what’s working well for other people. But no benchmark score guarantees performance on your specific task, especially once that task looks different from the benchmark’s own question format — and in some documented cases, benchmark scores are inflated because the test questions leaked into a model’s training data in the first place.
What MMLU and HumanEval Actually Measure
- 1
MMLU (Massive Multitask Language Understanding). Just under 16,000 multiple-choice questions spanning 57 subjects — mathematics, history, law, medicine, computer science, and more. A model picks one of four answer options per question; the score is simply the percentage answered correctly. High MMLU performance is a reasonable proxy for broad, textbook-style knowledge.
- 2
HumanEval.164 hand-crafted programming problems, each with a function signature, a docstring describing the task, and unit tests to check correctness. The model writes code; a solution “passes” if it satisfies the tests. It’s a genuinely good measure of short, self-contained coding ability — the kind of problem you might see in a coding interview.
Sources: Hendrycks et al., “Measuring Massive Multitask Language Understanding” and OpenAI, “Evaluating Large Language Models Trained on Code”.
Using Ollama’s Library Page Directly
Before reaching for an external leaderboard, Ollama’s own model library gives three practical, immediately usable signals: the exact parameter sizes available for a model family (so you know precisely what you’d be downloading), how recently it was updated (newer isn’t always better, but a stale, long-unmaintained model is a real yellow flag), and pull count — how many people have downloaded it. Pull count isn’t a quality benchmark, but it’s a real signal that a model is proven useful for common, everyday use cases, which matters when you don’t have time to evaluate every option yourself.
Where Benchmarks Mislead
A benchmark score is only as predictive as how closely your real task resembles the benchmark’s own question format. There’s also a documented, more concerning failure mode: benchmark contamination, where the actual test questions — or near-duplicates of them — end up inside a model’s training data, inflating its score without reflecting genuine capability. Researchers have found cases of MMLU questions appearing verbatim in web-scraped training data, and this risk grows for older, widely-published benchmarks that have had years to be absorbed into newer training sets.
See how the gap between benchmark rank and real-task performance actually plays out:
Fun Fact
MMLU was introduced in 2020 with roughly 16,000 questions — old enough, and public enough, that some of those exact questions have since been found sitting in later models’ training data, which is exactly why newer, contamination-resistant benchmarks keep appearing to replace it.
Test Yourself
What does the MMLU benchmark primarily measure?
What does HumanEval measure?
What is "benchmark contamination"?
Why might a model that tops a coding benchmark still perform poorly on a real coding task?
Ready to build your own, contamination-proof evaluation — one based on your actual task, not a public leaderboard? Continue to the Advanced level →