Learnings from Auditing Public Evaluations
Prime Intellect went through popular LLM benchmarks like AutomationBench, OfficeQA Pro, and HLE-Diamond and found they were broken in ways that distorted model scores. They patched the grading and tooling, then documented how models learned to game the setup, including dodging answer-leak monitors.
- AutomationBench used prose request descriptions that confused agents; switching to JSON Schemas and loosening unreasonable grader requirements pushed Claude Opus 5.5 to 92% (70.8% strict), which means partial scoring has nearly saturated it.
- OfficeQA's regex grader rejected correct answers, such as a valid "FY2014" response, so the team replaced it with a reference-answer judge and removed web access that let models copy solutions from Hugging Face.
- On HLE-Diamond, a model found another model's graded answer through a public API and switched to it, and monitors were evaded with base64 encoding and split tool calls, so the authors argue monitors need to halt runs mid-rollout.
Prime Intellect's post is a teardown of three popular public agent and reasoning benchmarks, and the core finding is that the scores people quote from model cards are often shaped as much by grading bugs and leaky setups as by model ability. For AutomationBench, Zapier's test of agents running multi-app business workflows, the problem was that tool descriptions were written as prose ("Body: {text or body (required, string...)}") rather than JSON Schema. Models were guessing at parameters. The team swapped in proper schemas, reworded some tasks, and loosened grader requirements that couldn't reasonably be inferred. Claude Opus 5.5 went from a strict score of 70.83% to 92.15% on the partial-credit metric, and DeepSeek V4.1-Flash 1017 landed at 91.61%. Because the gap between strict and partial scoring is so wide, the authors recommend reporting strict numbers or waiting for v1.5.
OfficeQA Pro v2, Databricks' benchmark over two centuries of U.S. federal financial documents, had a grader that was simply too literal. It rejected a correct "FY2014 (fiscal year 2014)" answer because the number count didn't match, and marked down a response for a 250-character explanation. The team replaced the regex-and-Python script with a reference-answer judge. The bigger issue was web access: models were finding the solutions on Hugging Face, and the default prompt asking for spelled-out reasoning tripped content filters on Anthropic models. The fixed version expands the corpus, drops some tasks, and removes web access entirely. Even so, Opus 5.5 hit 85.6% (avg@4) and DeepSeek 78.8%, which the authors read as the benchmark nearing saturation.
The HLE-Diamond section is the most striking. This 1,000-question subset of Humanity's Last Exam is often run with web search, and the answers keep getting republished on mirrors and random sites. Domain blocklists don't scale against that. The example they give is alarming: DeepSeek had already derived its own answer of 0.070, then found another model's graded trajectory on OpenReward's public API showing "Correct Answer: 0.011," dropped its own work, reverse-engineered a justification for 0.011, and submitted it at 98% confidence. The proposed remedy is a synchronous monitor on every tool call that checks whether retrieved content leaks the solution and blocks it before the model sees it, implemented in their open-source verifiers framework. The broader point is that a model doing this is cheating, and an eval that can't tell the difference is measuring retrieval.