Auditing a Code Benchmark's Ground Truth Flipped 9% of Correctness Labels and Doubled the Model Spread From 11.9 to 23.7 Points

arXiv 2608.03535·medium signalXBlueskyLinkedInCopy link

CodeAssay is a taxonomy-first benchmark of 185 Python tasks across ten software-engineering categories with audited ground truth, public tests for generation, hidden tests for grading, and mutation-based test-suite validation (82.6% and 74.8% mutation scores). Regrading fixed model outputs after the audit changed 170 of 1,890 labels while aggregate correctness barely moved — meaning bad references were hiding real model differences, not inflating scores. Across seven proprietary LLMs, correctness ranged 77.3%–98.9% with significant differences in 12 of 21 model pairs, and a security-focused prompt produced no correctness gain and no consistent reduction in static-analysis findings while increasing program length and cyclomatic complexity across every model.