AI Agent Benchmarks Got Gamed to Near-Perfect Scores Without Solving a Single Task

An AI agent benchmark score is supposed to answer one question: will this agent actually do the job. In April 2026, a UC Berkeley team showed that eight of the field's best-known AI agent benchmarks could be pushed to near-perfect scores without answering that question at all. A separate study, using no cheating whatsoever, found nearly the same blind spot: enterprise agents that looked identical on a leaderboard differed by a full order of magnitude once cost and reliability were actually measured.

What an AI agent benchmark score is supposed to prove

When a paper or a product page reports that an agent scored 91% on SWE-bench Verified, the number stands in for something specific: the agent read a real GitHub issue, wrote a patch, and the patch made a failing test suite pass. Extend that claim outward, to something like "an agent this capable will handle your codebase," and the extension depends entirely on the first claim being true.

Agent benchmarks proliferated through 2025 and into 2026 for a reasonable purpose: model quality alone stopped predicting whether an agent could complete a multi-step task with real tools, a real filesystem, and a real terminal. SWE-bench, WebArena, and their peers were built to close that gap by scoring agents on outcomes rather than answers. The two studies below tested whether the scoring itself could be trusted, and answered from two different directions.

In April 2026, a research team at UC Berkeley's RDI center checked whether it was. They built an automated agent whose only goal was to score well on eight of the field’s most cited agent benchmarks: SWE-bench Verified, SWE-bench Pro, WebArena, Terminal-Bench, FieldWorkArena, OSWorld, GAIA, and CAR-bench. It never solved a task in the sense the benchmarks intended. It scored between 73% and 100% on all eight anyway.

Eight AI agent benchmarks, eight exploits, zero tasks solved

The team, Hao Wang, Qiuyang Mang, Alvin Cheung, Koushik Sen, and Dawn Song, didn't attack the model. They attacked the harness around it: the scaffolding that hands a task to an agent, watches what it does, and decides whether it passed. Every benchmark they tried had some version of the same flaw. The evaluation environment leaked information or control that a genuinely task-solving agent was never supposed to have.

None of these exploits touched reasoning ability. They touched the wall between the agent and the answer key, and that wall turned out to have a door in it on every benchmark the team tried. The SWE-bench case is the clearest illustration: the benchmark scores a submission by running the repository’s own test suite after the patch is applied. Editing ten lines inside conftest.py, the file that configures how those tests run, made the suite report a pass regardless of what the patch actually contained. The agent never touched the bug. It touched the referee.

The researchers’ own fix list is unglamorous: isolate agents from the evaluation environment completely, never distribute reference answers with task configs, avoid running eval() on untrusted input, sanitise anything fed to an LLM judge, and adversarially test a benchmark before publishing it. None of that is a research breakthrough. It is closer to a code-review checklist that the field skipped while it was busy publishing leaderboard numbers.

A second study found the same blind spot without any cheating

A different research effort, proposing an evaluation approach called CLEAR (Cost, Latency, Efficacy, Assurance, Reliability), evaluated six agent architectures on 300 realistic enterprise tasks spanning customer support, data analysis, process automation, software development, compliance, and multi-stakeholder workflows. No exploit, no adversarial harness. The agents were simply run straight, then measured on more than the single accuracy column a leaderboard reports.

Two numbers stand out. Agents that scored well in a lab-style setup lost 37% of that performance once moved onto the enterprise tasks: lab benchmark scores overstated production readiness by more than a third. And agents reaching comparable accuracy differed by up to 50x in what it cost to get there, from $0.10 to $5.00 per task. The researchers found accuracy-optimised configurations cost 4.4 to 10.8 times more than a cost-aware, Pareto-efficient agent hitting similar accuracy on the same 300 tasks.

Put the two studies side by side and a pattern emerges that neither paper states outright: a benchmark score answers a narrower question than the industry has been treating it as answering. The RDI work shows the number can be manufactured without the harness noticing. The CLEAR work shows that even an honestly earned number leaves out cost, consistency, and domain fit, three variables that determine whether an agent is usable at all once it leaves the lab.

Reliability collapses the moment you run a task twice

Both papers converge on a variable a single benchmark run erases entirely: consistency. On the CLEAR tasks, single-attempt success ranged 68-74%. Repeat the same task eight times, a metric researchers call pass@8, and success dropped to 52-73%. One agent’s consistency fell from 72.3% on a first attempt to 58.3% by the eighth run, a 14-point slide that a one-shot leaderboard score cannot show.

For a team evaluating an agent framework or vendor, this is the number that changes a purchasing decision more than any exploit does. An 8-point drop across repeated runs might be tolerable for a low-stakes internal tool. A 14-point drop, the size the CLEAR study actually recorded for its weakest agent, means the tool that impressed everyone in the demo will fail on close to two out of every five repeats of that exact same task by Friday.

Cost hides behind the same accuracy number

The CLEAR paper’s highest-accuracy configuration, built on a Reflexion-style architecture, cost 5.12 times more than alternatives reaching comparable results. That gap doesn’t show up anywhere in a standard leaderboard, because leaderboards report a single accuracy column and stop there. A vendor quoting a benchmark score without a cost-per-task figure next to it is quoting half the number that matters.

What to check before trusting an AI agent benchmark score

Neither paper argues for abandoning agent benchmarks. Both argue that a bare accuracy percentage was never load-bearing enough to hold the claims built on top of it. Before treating one as evidence:

• Ask whether the harness isolates the agent from the answer key entirely, not just “sandboxes” it. Every RDI exploit worked inside an environment that was already called sandboxed.
• Ask for pass@k at k=8 or higher, not pass@1 alone. A single successful run tells you the ceiling, not the average day.
• Ask for cost per task alongside accuracy. A number that reports only percentage correct is hiding at least one axis the CLEAR study found 50x variance in.
• Ask whether the score came from an independent benchmark or the vendor’s own, and whether that benchmark’s harness has been adversarially tested before publication.
• Expect a domain-specific spread. The CLEAR paper measured 15-25% accuracy swings for the same agent just from switching enterprise task categories, so one average obscures more than it reveals.

None of these checks require re-running anyone else’s research. They mostly require asking the vendor or the paper for numbers that a well-run evaluation already has sitting in a spreadsheet, and treating a refusal to share them as itself an answer.

The harnesses will get harder to game, and CLEAR-style multi-dimensional scorecards are a reasonable bet for what vendor disclosures start to look like within the next reporting cycle. Until then, the number on the leaderboard is a starting question, not an answer.