Evaluation & Benchmarks
How to Evaluate LLMs: Benchmarks, Arenas, Agent Tests and Data Contamination
Intermediate 20 min For: Beginners / Developers / Researchers
Key takeaways
- ▸A benchmark is a fixed set of tasks plus a scoring rule; it measures only what its tasks cover, under its own conditions.
- ▸Knowledge/reasoning tests (MMLU, GPQA), coding tests (HumanEval, SWE-bench), agent tests (OSWorld) and human-preference arenas answer different questions.
- ▸Scores can mislead through data contamination, benchmark saturation, different prompting/scaffolding and optimising for the leaderboard itself.
- ▸For real decisions, build a small evaluation set from your own tasks and check it with clear criteria — automated where possible, human where it matters.
What a benchmark is — and isn't
A benchmark is a fixed set of tasks with a scoring rule: multiple-choice accuracy, unit tests passed, tasks completed. It tells you how a model (or a model plus its scaffolding) performs on those tasks under those conditions — nothing more. Reading leaderboards well means knowing what each benchmark contains.
The main families
| Family | Examples | What it measures | Watch out for |
|---|---|---|---|
| Knowledge & reasoning | MMLU (57 subjects, multiple choice), GPQA (graduate-level "Google-proof" science), Humanity's Last Exam | Breadth and depth of knowledge, exam-style reasoning | Saturation (top models near ceiling on MMLU), contamination |
| Maths | GSM8K, MATH, competition sets (AIME) | Multi-step quantitative reasoning | Leaked problems; answer-only scoring |
| Coding | HumanEval (164 Python functions), SWE-bench (real GitHub issues, judged by the repo's tests) | Writing correct code; fixing real repositories | Different agent scaffolds give very different scores |
| Agents & computer use | OSWorld (369 real desktop tasks), τ-bench, WebArena | Completing multi-step tasks in real software environments | Environment setup, time limits, flaky tasks |
| Human preference | Chatbot Arena / LMArena | Which answer people prefer in open chat | Style and length bias; selective testing |
| Holistic suites | HELM | Many scenarios and metrics (accuracy, calibration, robustness, bias) | Breadth over depth |
Some useful anchors from the original papers: SWE-bench contains 2,294 issue–fix pairs from 12 popular Python repositories (Jimenez et al., 2023); because some tasks were ambiguous or untestable, OpenAI and the authors released SWE-bench Verified, 500 tasks checked by human developers (OpenAI, 2024). When OSWorld was published, humans completed 72.36% of its tasks while the best model completed 12.24% (Xie et al., 2024) — agent benchmarks expose how far models are from reliable computer use.
How arenas work
In Chatbot Arena (Chiang et al., 2024) a user sends a prompt to two anonymous models and votes for the better answer. A Bradley–Terry model (the statistics behind Elo ratings) converts many pairwise votes into scores with confidence intervals. Arenas capture real user preferences on open-ended prompts, but preferences reward style, formatting and length, and analyses such as The Leaderboard Illusion (Singh et al., 2025) argue that private pre-release testing and unequal sampling can distort rankings.
Why scores mislead
- Contamination – test items leak into web-scale training data, inflating scores through memorisation.
- Saturation – once top models score near 90%+, a benchmark stops separating them; new, harder sets appear (GPQA, Humanity's Last Exam).
- Setup differences – prompting, number of attempts (pass@1 vs pass@k), tool access and agent scaffolding can change results by many points. Compare like with like.
- Goodhart's law – "when a measure becomes a target, it ceases to be a good measure." Optimising for a leaderboard can produce models that are better at the benchmark than at the job.
- Model vs system – agent benchmarks score the whole system: model, tools, prompts, retries and environment (How AI Agents Work).
flowchart LR Q["Your question: which model/system for my task?"] --> P["Public benchmarks: shortlist candidates"] P --> O["Own eval set: 30–200 real tasks with clear pass criteria"] O --> A["Automatic checks (tests, exact match, schema)"] O --> J["LLM-as-judge, validated on human labels"] O --> H["Human review for high-stakes cases"] A & J & H --> D["Decision + regression suite for future updates"]
Evaluating for your own use
- Shortlist with public benchmarks that resemble your task.
- Collect real examples of the task (30–200 is a good start) and write down what a correct result is.
- Score automatically wherever possible: unit tests, exact match, JSON-schema validation, retrieval recall (RAG).
- Use LLM-as-a-judge for open-ended outputs, but measure its agreement with human labels first; judges show position and verbosity biases (Zheng et al., 2023).
- Keep it as a regression suite: re-run whenever you change the model, prompt or tools.
More hands-on detail is planned in Building Evals for Your Own AI App and Evaluating Agents (in progress).
FAQ
What is the best LLM benchmark?
There is none for all purposes. Use benchmarks that resemble your task: SWE-bench Verified for coding agents, OSWorld for computer use, GPQA or Humanity's Last Exam for expert knowledge, and arena-style preference rankings for general chat quality — then test on your own data.
What is benchmark contamination?
When test questions (or their answers) appear in a model's training data, so a high score may reflect memorisation rather than ability. Because web-scale training data is hard to audit, newer benchmarks use fresh, private or regularly refreshed questions.
How does Chatbot Arena rank models?
Users chat with two anonymous models side by side and vote for the better answer; a statistical model (Bradley–Terry, similar to chess Elo) turns millions of pairwise votes into a ranking with confidence intervals.
Can an LLM judge another LLM's answers?
Yes, “LLM-as-a-judge” is widely used and agrees with humans reasonably often, but judges have biases (answer position, length, their own style). Validate a judge against human labels before trusting it.