Fetching from the wire…
01
02
03
04
05
06
07
08
Research2026-08-06 · source-backed
arXiv 2608.04975 is a domain-expert audit of all 65 SciCode problems, a component of the Artificial Analysis Intelligence Index and a standing eval in government and national-lab suites. 192 of the 263 defects, spread across 91% of main problems, wrongly rejected correct solutions via non-reproducible gold answers, over-tight tolerances, or self-contradictory specs. 78% of score-suppressing defects required specialized physics or math to spot, not proofreading. Re-evaluating twelve frontier snapshots on the corrected set lifts subproblem accuracy from 45-60% to 84-98% and main-problem accuracy from 9-27% to 69-92%. The widely-cited 2026 scientific-coding plateau was the instrument.
Each link below shares sources, entities, or timing with this story.
Shared entity: Artificial Analysis Intelligence Index / Shared topic / Earlier coverage
Both cover Artificial Analysis Intelligence Index; overlapping topics (analysi, artificial); earlier Artificial Analysis Intelligence Index coverage from 2026-03-21.
Same source domain / Shared topic / Downstream implication
Reported by the same outlet (arxiv.org); overlapping topics (eval, frontier); traces where this leads (implication).
Same source domain / Shared topic
Reported by the same outlet (arxiv.org); overlapping topics (analysi, audit, eval).
Reported by the same outlet (arxiv.org); overlapping topics (accuracy, answer, correct).
Reported by the same outlet (arxiv.org); overlapping topics (accuracy, answer, defect).
Shared entity: Artificial Analysis Intelligence Index / Earlier coverage / Tension
Both cover Artificial Analysis Intelligence Index; earlier Artificial Analysis Intelligence Index coverage from 2026-04-21; pushes against this story (versus).
Shared topic / Tension
Overlapping topics (accuracy, answer, correct); pushes against this story (vs).
Overlapping topics (analysi, artificial, eval); pushes against this story (vs).