Real-SWE benchmarked coding agents on private production code, and the best score is 38.8%

Every number you use to pick a harness comes from public repositories the models may have trained on. Specific Labs built the version that doesn't: tasks drawn from licensed private company codebases, including a 200K-user event app and a fintech processing over 100K bank statements. Each model runs inside its maker's own harness, pass@1 averaged over eight runs, median task editing 11 files.

Claude Fable 5.1 in Claude Code leads at 38.8%. GPT-6 Astra in Codex CLI takes 33.8%. Then Gemini 3.8 Flash at 31.2%, GLM 5.3 at 28.8%, Grok 4.6 and Muse Spark 1.3 tied at 23.8%, Kimi K3 at 18.8%, GPT-5.6 Sol at 16.2%. 248 points on HN.

Sit with the top number for a second. Six out of ten tasks unresolved, by the best model in the best harness, on code nobody published. The public SWE-bench Verified figures people quote in procurement decks run 70-80%. Some of that gap is task difficulty and some is memorization, and this benchmark can't separate them cleanly. What it can do is tell you the out-of-distribution floor, because private code is natively out of distribution and stays that way.

The cost line turns it into an argument you can take to a finance conversation. Fable runs $6.96 per rollout, which works out to about $17.94 per resolved task at 38.8%. Astra's lower pass rate means a worse cost-per-resolution even where the per-rollout price is competitive. That's the metric to build your own dashboard around, and it matches what AWS found on a different axis this week, where a pricier model came out 8x cheaper per correct answer.

Two caveats I'd state out loud. Model-in-own-harness is the right call for deciding what to buy and the wrong call for isolating model quality, because you're measuring the pair. And "licensed private codebases" means you can't reproduce it, which is exactly the property that makes the tasks uncontaminated. You're trading verifiability for validity. I'll take that trade over a leaderboard the training set has seen, but know which one you're holding.

The action is unglamorous: build a ten-task internal benchmark from your own closed issues, run it pass@1 over eight attempts against two harnesses, and record dollars per resolved task. It takes an afternoon and it will disagree with the public leaderboards. Mine did.