Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-25 · high
GPT, Gemini, Claude and Grok show a 0.075 correlation between passing C# tests and writing clean code.
Why now: The paper posted to arXiv in August 2026, filling a gap in a benchmark field built almost entirely around Python.
A new evaluation framework scored GPT, Gemini, Claude and Grok on 85 C# programming problems, and passing the tests barely predicted clean code.
The correlation between functional correctness and static code quality came out to r = 0.075, close to zero. Leaderboards that rank models by test pass rate don't show whether the code is maintainable, because the two scores move independently.
Researchers ran four models against 85 algorithmic tasks adapted from HumanEval, producing 340 solutions, then scored each one on three separate axes: unit tests for correctness, Roslyn AST analysis for code quality, and adversarial BenchmarkDotNet profiling for runtime efficiency. The full setup is in the benchmark paper.
Most coding benchmarks live in Python. Enterprise .NET shops picking a model for C# work have had almost nothing to check against.
A pass rate on unit tests tells you whether the code runs, not whether you'd want to maintain it. A model that aces the tests can still write code that's a mess to read, and a model with a lower pass rate can still write the cleaner solution. Watch whether future benchmarks start reporting quality and efficiency scores alongside pass rate instead of just correctness.
Each link below shares sources, entities, or timing with this story.
Gemini competes with Claude / Shared entities / Same source domain / Earlier coverage / Tension
Linked by a graph relationship (Gemini competes with Claude); both cover Claude, Gemini, GPT, LLM; reported by the same outlet (arxiv.org).
Gemini competes with Claude / Shared entities / Same source domain / Earlier coverage
Linked by a graph relationship (Gemini competes with Claude); both cover CLAUDE, GPT, LLM, Pass; reported by the same outlet (arxiv.org).
Linked by a graph relationship (Gemini competes with Claude); both cover CLAUDE, Gemini, GPT, LLM; reported by the same outlet (arxiv.org).
Gemini competes with Claude / Shared entities / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Gemini competes with Claude); both cover CLAUDE, Gemini, GPT; overlapping topics (adversarial, automated, claude).
Cursor uses Grok / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Cursor uses Grok); both cover Claude, GPT, Grok; overlapping topics (claude, code, enterprise).
Gemini competes with ChatGPT / Shared entities / Earlier coverage
Linked by a graph relationship (Gemini competes with ChatGPT); both cover Claude, Gemini, GPT, Grok; earlier Claude coverage from 2026-06-21.
Cursor uses Grok / Shared entities / Earlier coverage
Linked by a graph relationship (Cursor uses Grok); both cover Claude, Gemini, GPT, Grok; earlier Claude coverage from 2026-06-19.
Gemini competes with Claude / Shared entities / Same source domain / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Gemini competes with Claude); both cover CLAUDE, LLM; reported by the same outlet (arxiv.org).