Accounting benchmark: no AI model tops 2.6% pass rate

Mercor and Ramp built an accounting benchmark graded by real accountants, and no model cleared a 2.6% Pass@8 completion rate, per the paper on arXiv.

That gap is the number to sit with if you're weighing whether an agent can close your books unsupervised. The benchmark's best rubric score, 56.4%, sits nowhere near a rate you'd trust without a human checking the work.

The benchmark, called APEX-Accounting, spans 160 private tasks across 10 self-contained worlds, each with its own accounting system, spreadsheets, and PDFs. Practicing accountants wrote every task, solved it themselves, and built the grading rubric, so there's no ambiguity about what counts as done.

On that rubric, Claude-Fable-5 running at Max effort led with 56.4% Mean Criteria@3. Muse-Spark-1.1 at xHigh effort followed at 52.6%. Check Pass@8, the rate at which a model finishes a task well enough to actually use, and both scores collapse to under 2.6%.

The paper also documents a Simpson's paradox in token spending. Raise the budget from $1 to $50 per task and the aggregate score climbs, but individual tasks often score worse with more tokens spent. APEX-Accounting doesn't say why. My read: more tokens just give a model more room to talk itself out of a right answer.

56.4% on a rubric means the model understood the assignment. It doesn't mean the ledger balances. Watch whether the next accounting benchmark leads with Pass@8 instead of rubric credit. That's the signal that builders took this gap seriously.