Best AI model reads scientific papers at 62.6 percent

A new benchmark scored the best multimodal AI system at just 62.6 out of 100 on real scientific reading tasks, per SciDocBench. Document perception, evidence grounding, verification, and cross-document reasoning came in as the weakest categories. Those four skills are what a research assistant is supposed to do: read a paper, pull the right evidence, and check its own answer.

SciDocBench draws on 124 expert-authored questions, screened for difficulty, across seven capability groups and 19 subtasks in five domains. Each question ran under four conditions, English or Chinese, images shown all at once or interleaved with text, producing 496 test instances total.

The systems fell apart in four specific places: reading documents accurately and grounding claims in the right evidence. They also struggled to verify their own answers and reason across more than one document.

Alongside the benchmark, the paper released SciDocIR, a typed evidence-graph format. It preserves document objects, layout, cross-references, and provenance instead of flattening everything into plain text. The release includes about 15,000 supervised training examples built on that structure.

The benchmark doesn't say which architectural fix closes the gap. It just gives the next claim of "we read the paper" something to be measured against.