"Which model is best?" You open a leaderboard, check who's on top this month, and sign. It's fast, it's quantified, it feels safe.
The problem: those leaderboards don't measure what you think. They don't tell you whether a model will do your work properly - they tell you who aced a public exam whose past papers everyone has already seen.
Good news: the evaluation that matters is one you can build yourself, and it takes a day.
A score you optimise for stops being a measure
The evidence piled up fast. Hugging Face, which hosted the best-known open-model leaderboard, eventually archived it: after two years and tens of thousands of submitted models, teams were optimising for that specific leaderboard rather than for general capability.
An April 2025 study, "The Leaderboard Illusion", describes another flaw in the popular Chatbot Arena comparison platform: some major labs reportedly test large numbers of unreleased variants there quietly - the authors cite up to 27 variants before a single model ships - and only make the best one public. At that point the leaderboard no longer compares models, it compares best attempts.
Even a demanding evaluation can be compromised without any bad intent: Epoch AI had to state publicly that OpenAI, the funder of its FrontierMath maths benchmark, had access to most of the problems and their solutions.
And we now know cheating can be accidental. Researchers have shown that simply including slightly rephrased test questions in the training data is enough to gain tens of points - with zero real improvement in capability.
The right image is cramming. Hand a student the past exam papers and they'll score 18/20 without understanding the subject any better - and you learn nothing about how they'll handle their first real case. That's exactly what happens when a score becomes a commercial target: it stops being a measure.
What a benchmark actually is
Technically, a benchmark isn't a grade, it's three things. A frozen question set (often a few hundred items, multiple choice or with a verifiable answer). A test harness: how the instruction is phrased, how many attempts are allowed, how the answer is read. And an aggregation into a single percentage.
All three are fragile. The question set leaks into training data (that's "contamination"). The harness varies from lab to lab - which is exactly why OpenAI open-sources simple-evals, to make these protocols reproducible. And the final percentage crushes everything else: cost, latency, reliability, behaviour when the model gets it wrong.
Then add saturation. GPQA Diamond, designed in 2023 to resist web search, went from roughly 39% to over 80% in under two years. Stanford's AI Index 2025 notes that reference tests saturate faster and faster, sometimes in less than a year - hence ever-harder exams like Humanity's Last Exam (2,500 questions, under 10% success at launch). A saturated test separates nobody.
The measures that actually look like work
Not everything should go in the bin: some approaches get closer to real usage, and they're worth knowing.
- Task duration. Instead of a question-and-answer score, METR measures the length of tasks a model completes half the time - and finds that this duration roughly doubles every 7 months. That's already closer to your real question: "how far can it go alone before I have to take over?"
- Human voting. Chatbot Arena has humans vote on head-to-head answer duels (over a million votes at the time the study was published). Useful for general feel, silent on your business.
- Cleaning up the tests. OpenAI released SWE-bench Verified - 500 tasks manually re-checked out of the original 2,294 - after finding that a notable share of tasks were badly specified, or even unsolvable.
- Like-for-like comparison. Independent observers such as Artificial Analysis standardise prompts, infrastructure and metrics, and show quality alongside price, speed and latency. That's the right way to read a leaderboard: never a score on its own.
The only benchmark that counts is yours
Anthropic puts it well in its work on evaluation: you need evaluations specific to the deployment context, not a single score.
In practice it's very doable. Take 30 to 50 cases drawn from your actual business: typical customer requests, awkward documents, three known traps. For each one, write down what a good answer looks like. That's your reference set - your "gold". Then: replay that set on every new model, on every prompt change, and record not just accuracy but also cost per task and turnaround time. Keep those cases private: the day they end up in a public repo, they become contaminable, like all the others.
And remember where most of the gap comes from. With the same model, it's what surrounds it - the loop, the tools, the memory - that decides the outcome. A leaderboard grades a bare engine; you're evaluating a whole car on your own road. That's also what makes "which model is best?" a secondary question: the model is a commodity, and an in-house evaluation lets you switch overnight, evidence in hand.
What we do
At Elephant in the Room, we install this evaluation loop at our clients': a set of real cases drawn from their business, plugged into their company brain, replayed automatically on every model or prompt change, with quality, cost and latency side by side - and a human deciding what matters. That's what lets you pick a model on your numbers, not on the ones in a press release. And it's how we work ourselves: our own agents are evaluated on our own cases. We don't sell a theory we don't apply.
A leaderboard tells you who passed the exam best. Your own cases tell you who can do the job.
