Physics faculty re-graded six benchmarks, and GPT-5.6-Sol went from 47.3% to 78.7%

This one should change how you read leaderboards.

A physics benchmark audit put faculty and graduate researchers through six widely used physics benchmarks, including ones feeding the Artificial Analysis Intelligence Index that half the industry quotes. They reviewed problem statements, reference solutions, and model responses, separating genuine reasoning failures from three kinds of benchmark defect: grader errors, wrong reference solutions, and underspecified questions.

Most cases originally scored incorrect were benchmark defects. After correction, GPT-5.6-Sol's mean@4 on HLE-Physics rises from 47.3% to 78.7%. On CMT-Benchmark, 61.0% to 87.2%. Corrected pass@4 on 54 retained CritPt challenges: 94.4%. UGPhysics, PRISM-Physics and PHYBench audited subsets all rise substantially too.

A 31-point correction isn't noise around the edges. It means the reported number was measuring the benchmark's quality more than the model's.

The same day, Dan Luu published his dissection of Senior SWE-Bench and found the same failure from a different angle. He re-ran the LLM grader ten times on identical solutions and 23% of official results flipped. The headline rankings (Fable 5 at 29.1%, Opus 4.8 at 25.0%, GPT-5.6 Sol at 24.4%) sit inside a scoring scheme with discontinuous thresholds, including one task whose 1-line reference solution forces tasteful three-line answers into the wrong bucket. Each task runs exactly once despite large per-run variance. Only 4 of 113 tasks are Rust.

Two independent audits, two different domains, same conclusion: the grader is a bigger source of variance than the models being compared.

The instruction that falls out of this is specific. Stop picking models off leaderboard deltas smaller than the grading noise. For SWE-Bench-family results that means anything under about five points is unreadable, since a 23% flip rate on re-grading swamps a 4.7-point gap. For physics indices, the entire ordering should be considered provisional until somebody re-audits.

Luu's Rust observation deserves its own note, because it collides with Guillermo Rauch's claim this week that language choice on human convenience grounds is over now that agents write the code. Maybe. But the benchmarks everyone uses to decide which agent to trust barely test Rust at all, so the evidence for "agents are equally good across languages" doesn't exist yet in the place people are looking for it. Rauch is describing what he sees inside Vercel. That's one company's internal velocity, and he's honest about it being an observation.

I don't know how to fix this as an individual builder. I can't re-audit HLE-Physics. What I can do is stop treating a 2-point leaderboard move as information, and run my own eval on my own tasks before switching models. The 30 minutes that costs is cheaper than picking wrong off a broken scoreboard.