ExtractBench: 4,869 pages of enterprise documents expose that VLM agents silently truncate long record lists

arXiv·medium signalXBlueskyLinkedInCopy link

Released July 31 (arXiv 2607.29677) by a team including Adrian Lyjak and Simon Suo, ExtractBench evaluates schema-guided extraction — give an agent a document and a user-defined schema, get back correct values plus source evidence — across 370 enterprise documents, 4,869 pages, 8 business domains and 67 document types. It scores three separate dimensions: order-insensitive value F1, word-level grounding F1, and page-level grounding F1, so an agent cannot fake traceability. The headline failure mode is specific and actionable: commercial vision-language models do well on short documents but often truncate record lists on long ones, while coding agents hold accuracy at substantially higher cost. Dataset and eval code are on HuggingFace and GitHub.