Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-03 · high
Coding agents keep accuracy on the same documents, but at substantially higher cost, per the ExtractBench paper.
Why now: Coverage of the paper as of Aug. 3 already includes the dataset and eval code, giving builders a way to test their own extraction setup before shipping.
Commercial VLMs silently truncate record lists on long documents, per a new benchmark called ExtractBench.
That's the exact failure mode for anyone piloting AI extraction on long, multi-page filings. The model can look accurate while quietly dropping records, and a normal accuracy score won't catch it.
The paper, from Adrian Lyjak, Simon Suo and co-authors, tests schema-guided extraction on 370 enterprise documents across 4,869 pages, 8 domains and 67 document types. It scores value accuracy, word-level grounding and page-level grounding separately, so a model can't fake traceability by getting the right answer without showing its work.
On short documents, the VLMs it tested score well. Push the record count up and they start truncating without flagging that anything was cut.
Coding agents hold accuracy on the same longer documents, but run at substantially higher cost. The paper doesn't say by how much, just that the gap is substantial.
Dataset and eval code are public on HuggingFace and GitHub.
Each link below shares sources, entities, or timing with this story.
HuggingFace released Claude Code / Shared entity: GitHub / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (HuggingFace released Claude Code); both cover GitHub; overlapping topics (agent, code, coding).
Linked by a graph relationship (HuggingFace released Claude Code); both cover GitHub; overlapping topics (agent, code, coding).
Linked by a graph relationship (HuggingFace released Claude Code); both cover GitHub; overlapping topics (agent, code, coding).
Linked by a graph relationship (HuggingFace released Claude Code); both cover GitHub; overlapping topics (code, coding, cost).
Linked by a graph relationship (HuggingFace released Claude Code); both cover GitHub; overlapping topics (agent, code, coding).
HuggingFace released Claude Code / Shared entity: GitHub / Shared topic / Earlier coverage
Linked by a graph relationship (HuggingFace released Claude Code); both cover GitHub; overlapping topics (agent, code, coding, cost).
HuggingFace released Claude Code / Shared entity: GitHub / Same source domain / Shared topic / Earlier coverage
Linked by a graph relationship (HuggingFace released Claude Code); both cover GitHub; reported by the same outlet (arxiv.org).
HuggingFace released Claude Code / Shared entity: GitHub / Shared topic / Earlier coverage
Linked by a graph relationship (HuggingFace released Claude Code); both cover GitHub; overlapping topics (agent, code, coding, cost).