Fixing 170 bad labels doubled a coding benchmark's spread

CodeAssay's audit flipped 170 of 1,890 model-output labels in a 185-task Python benchmark, per the paper's regrading results. Aggregate correctness across the benchmark barely moved. But the bad labels weren't neutral. They were hiding real gaps between models, and fixing them nearly doubled the measured spread between best and worst performers, from 11.9 points to 23.7.

CodeAssay builds its taxonomy first: public tests for generation, hidden tests for grading, and mutation testing to validate the references themselves. That design caught the 9% label error rate, the same fix behind the jump in spread. The paper doesn't say which individual rankings flipped, only that the aggregate gap nearly doubled.

The paper also ran a security-focused prompt against every model tested. It produced no gain in correctness and no consistent drop in static-analysis findings. What it did do, consistently, was make the generated programs longer and more cyclomatically complex. The paper doesn't say whether that added complexity came from real safety fixes or defensive boilerplate.