Expert re-grading found 60% of CMT-Benchmark questions defective, and one model's score nearly doubled.

Physics faculty audited six text-only physics benchmarks (arXiv 2609.13009) and found most cases scored as model errors were broken answer keys, underspecified problems, or graders rejecting equivalent correct answers. 30 of 50 CMT-Benchmark questions and 21 of 56 CritPt questions contained defects. After correction, GPT-5.6 Sol's mean@4 goes from 47.3% to 78.7% on HLE-Physics and 61.0% to 87.2% on CMT-Benchmark, with corrected pass@4 reaching 94.4% on the 54 retained CritPt challenges. A mid-40s score on a hard science benchmark may be measuring the grader.