GAMUT benchmark shows top AI model misses half of complete answers

A new benchmark called GAMUT scores AI models on something factuality tests have mostly ignored: whether an answer is complete, not merely whether it's correct. The paper, posted to arXiv, tested 14 frontier and open-weight models against 1,813 questions grounded in real wearable-device imagery across 10 domains, with evidence verified by experts. The best score, from Gemini 3.1 Pro, was 58.7 percent.

That number matters because factuality evaluation has almost entirely measured precision, whether the claims a model makes are true, while leaving completeness unmeasured. A model can state only accurate facts and still fail if it omits half of what a correct answer requires. GAMUT's method is a two-level meta-rubric: a structured rubric that captures how required content should be organized and weighted, then compiled into flat binary checks an LLM judge can grade consistently.

The gap between GAMUT's top score and 100 percent is the story. The strongest model in this test gets less than 60 percent of required content into its answers. That makes completeness a wide-open failure mode that precision-only benchmarks have been hiding, not a minor tuning problem.

What to watch

Worth watching whether other benchmark builders adopt a completeness axis alongside precision, and whether scores move much as models get optimized against it specifically. If GAMUT's approach holds up, expect current leaderboards to look different once completeness gets factored in. Builders shipping anything that summarizes or answers from images, wearable or otherwise, should treat a correct-sounding answer as no guarantee it's a complete one.