EarthVerse: best agent hits 84.65% answer-unit accuracy but only 34.81% Strict@95

arXiv·medium signalXBlueskyLinkedInCopy link

EarthVerse benchmarks scientific agents on 405 reproducible package-scoped investigations grounded in 199 documented events across 19 hazard families, requiring agents to choose compatible evidence, run transparent calculations, reconcile conflicting sources and preserve provenance. Executable ground truth decomposes each task into fine-grained answer units alongside rubrics that score the research process while allowing multiple valid paths. Across 25 model and agent systems the best mean answer-unit accuracy was 84.65% while the highest Strict@95 was 34.81%, the gap showing agents complete individual steps without holding a consistent chain across units, scales and physical interpretation.