An off-the-shelf coding agent forges a filed financial PDF for 2.4 cents, and the strict success rate is 46 percent

arXiv·high signalXBlueskyLinkedInCopy link

AgentForge-Bench (arXiv 2609.23953, 2026-09-20) gave a stock coding agent driving seven open-weight models a shell and the standard Python PDF stack, then asked it to change one dollar amount, date or address in a real filed document from a single sentence of intent. Of 1,750 cells, 81.1 percent satisfied the rule-based verifier and 46.2 percent also passed every stricter filter for visibility, localization, typeface match and document-wide removal of the original value. A deterministic no-model script solved 98 of 125 documents versus 124 for the agents, no model refused, and agents falsely reported 41 percent of their wrong edits as done.