Ignore the Leaderboards: Run Your Own AI Model Comparison

Every time a new AI model ships, the same ritual: benchmark charts, a leaderboard reshuffle, and a wave of posts declaring a new king. Then you try it on your actual work and it’s… fine? Sometimes better, sometimes worse, rarely in the ways the charts promised.

That’s not the benchmarks lying, exactly. It’s a mismatch between what they measure and what you need. Understanding why is the fastest route to the alternative, which is cheaper than you’d think.

Why leaderboards are a weak signal for you

They measure the wrong distribution. A benchmark is a fixed exam: competition math, trivia-style questions, standardized coding puzzles. Unless your job looks like that exam, a model can climb the chart without getting better at your work — and vice versa.

Teaching to the test is structural. Benchmark scores drive headlines, and headlines drive adoption, so labs optimize for them. That’s not scandal, it’s incentive gravity. But it means the gap between “scores well” and “works well” widens exactly where the marketing pressure is highest. And when test questions leak into training data — which happens, and is hard to fully prevent — a score can reflect memory rather than ability.

Averages hide the shape. A model that’s brilliant at code and mediocre at prose can post the same aggregate score as one that’s the reverse. You don’t work in aggregate. You work in your specific corner, and the chart doesn’t have a bar for it.

Vibes don’t fix this either. The popular alternative — crowd-sourced head-to-head voting — measures which answer strangers prefer at a glance. That rewards confident formatting and pleasant tone, which are real things, but they are not accuracy, and a glance can’t tell the difference.

None of this makes public benchmarks useless. They’re decent for spotting generational leaps and roughly tiering models. They’re just too blunt for the decision you actually face: which of these two or three candidates should I use for my work?

The one-evening personal eval

The fix is the same move I recommend for evaluating any AI tool: stop letting the product choose its own exam.

1. Collect ten real tasks. Pull them from your actual history — the last ten substantial things you asked an AI to do, or should have. Keep the originals: your real prompt, however messy. Don’t clean them up; the mess is part of what you’re testing.

2. Write the answer key first. For each task, note in one line what a good output must contain, and any instant-fail conditions (“invents a citation,” “breaks the API contract,” “exceeds the word limit”). Doing this before running anything is what keeps you honest later.

3. Run every candidate on every task. Same prompt, no coaching, no retries. Retries are a real feature in daily use, but round one should measure the default, because the default is what you’ll get at 4:50pm on a Friday.

4. Grade blind. Strip the model names, shuffle, then score against your answer key: 2 for shippable, 1 for salvageable, 0 for a rewrite. Blinding matters more than it feels like it should — knowing which model wrote an answer measurably changes how people rate it, and you are people.

5. Read the failures, not the average. The totals pick a winner, but the zeros tell you the shape: one model flops on long context, another on instruction-following, another invents details under pressure. Shape is what you actually use day to day — it tells you which model gets which kind of task.

6. Keep the harness. The ten tasks and the answer key are the durable asset. New model ships? Rerun the same ten, twenty minutes, done. You now have the thing no leaderboard can sell you: a longitudinal record of what works on your distribution.

What this buys you

Mostly, immunity. Immunity to launch-day hype, to cherry-picked demo reels, to the quiet regressions that never make a changelog. When someone insists the new model changes everything, you can answer with the only benchmark that was ever about you: “It went 14 out of 20 on my eval; the old one went 15.”

Ten tasks, one evening, rerun forever. It’s the least glamorous advice on this site and comfortably the highest return.