When a new artificial intelligence model is released, the announcement almost always arrives with a scoreboard: percentages on exams, rankings against rival systems, bar charts showing steady improvement. Those numbers shape billions of dollars in investment, influence policy debates, and tell the public how close machines are getting to human-level reasoning.
But as a recent Quanta Magazine profile highlights, a growing group of researchers argues that the way we measure AI is far less rigorous than the numbers suggest. One of them wants to change that by treating benchmarking as a science in its own right, with the same standards of evidence, error analysis and reproducibility that govern other empirical fields.
Why the numbers can mislead
A benchmark is simply a collection of tasks with a scoring rule. In principle, running two models on the same tasks should reveal which performs better. In practice, the exercise is riddled with complications.
Test questions can leak into the enormous training corpora scraped from the internet, so a model may effectively have seen the answers before the exam. Small changes in how a prompt is phrased can swing results. Scores are often reported as single numbers without confidence intervals, making it impossible to tell whether a gap between two systems reflects genuine capability or statistical noise. And because benchmarks are built by humans with particular interests, they capture a narrow slice of what a system can actually do, then get interpreted as broad statements about intelligence.
The result is a measurement culture that can feel more like marketing than science â a dark art in which insiders know which caveats to apply and everyone else reads the headline figure.
What a science of evaluation would look like
Researchers pressing for reform point to the fields that solved similar problems long ago. Psychometrics, the discipline behind standardized testing, spent a century developing theory about what a test score means, how reliably it can be estimated, and whether it generalizes beyond the specific items on the page. Metrology â the science of measurement itself â insists that every reported quantity come with an uncertainty attached.
Applied to AI, that approach would mean reporting error bars as a matter of course, running evaluations across many prompt variations rather than a single formulation, documenting exactly how a benchmark was constructed and scored, and checking whether test data overlaps with training data before trusting a result. It would also mean asking a harder question up front: what construct is a benchmark actually supposed to measure, and is there evidence it measures that thing rather than something correlated with it?
Stakes beyond leaderboards
This is not an academic quibble. Governments drafting AI rules increasingly reference capability thresholds, and companies cite evaluation results when arguing that a system is safe to deploy. If the underlying measurements are fragile, so is everything built on top of them. Benchmarks also steer research itself: labs optimize for whatever is being scored, so a flawed test does not just mislead observers, it reshapes what gets built.
The reform effort faces structural headwinds. Building careful evaluations is slow, unglamorous work, and the incentives in a competitive industry favor fast, flattering numbers. Benchmarks also saturate quickly, as models reach ceiling scores and the test loses its ability to discriminate.
Still, the argument is gaining traction. Dedicated evaluation research groups, benchmark-auditing projects and conference tracks devoted to measurement have all emerged in recent years. The goal is modest but consequential: when someone says a model scored 90 percent, everyone should know exactly what that means â and how much to trust it. Read More
Leave a Reply