Public leaderboards answer a question nobody building a real product actually asked: which model is best, on average, across a broad test set designed for comparability rather than relevance. Routing production traffic across Claude, OpenAI, and Gemini for four different products taught me that the leaderboard rank and the right choice for a specific task correlate a lot less than the marketing around each new release would suggest.
Why the public number misleads you
A leaderboard score is an average across tasks you don't run, weighted in a way you don't control, on a cutoff of training data you can't inspect. A model can lead the aggregate and still underperform specifically on your task (structured extraction from messy, even handwritten, input, say) because that narrow skill isn't weighted heavily in whatever benchmark produced the headline number.
The leaderboard tells you which model is generally strong. It cannot tell you which model is strong at the one thing you're actually going to ask it to do, millions of times, in production.
What an actual production benchmark needs
- Your real inputs, not a public test set. We evaluate the resume parser against genuinely messy, inconsistent, sometimes handwritten source documents, not clean sample resumes, because that's the actual distribution it has to survive.
- A task-specific scoring rubric, built with the domain expert who'd judge the output manually, not a generic correctness score that misses what "good" means for that specific task.
- Cost and latency in the same table as quality, because a 2% quality edge that costs four times as much per call, or takes three times as long, is not obviously the right production choice. It's a trade-off that needs to be made explicitly, by someone who owns the bill.
- Re-running the benchmark on every model update, because a quiet model refresh can change behavior on your specific task even when the public benchmark barely moves.
What this actually looked like
For the resume parser, the model that led public benchmarks at the time was noticeably worse on our specific task than a competitor. It over-corrected ambiguous handwriting into plausible-looking but wrong text, which is a worse failure mode for compliance documents than visibly failing to parse. The model we actually shipped with wasn't the aggregate leader. It was the one that failed more honestly on the inputs that mattered.
Across four products we now run a standing task-specific eval set per model family, re-scored on every update, which is also the backbone of the routing system that helped cut AI running costs 38% without a drop in output quality.
What I'd tell a team about to pick "the best model"
Stop asking which model is best. Ask which model is best at the narrow, specific, slightly boring task you're actually going to run millions of times, and build the five-input test that answers that, before you build the five-hundred-slide deck comparing leaderboard scores. The deck will be more impressive. The test will be the thing that's actually right.