Same Model, 17 Different Answers: The Scaffolding Gap Is Now Quantified

A systematic comparison of 15 AI coding agents running the same Claude Opus 4.5 model found that Augment, Cursor, and Claude Code produced a 17-problem spread on 731 SWE-bench Verified issues. Not different models. Not different prompting strategies visible to the user. The same underlying model, producing meaningfully different results based entirely on how each product wraps it — context engineering, tool orchestration, and harness design. LogRocket

This empirically confirms what practitioners have suspected: the scaffolding around the model matters as much as the model itself. When you see a leaderboard score for "Claude Opus 4.5 on SWE-bench," you're actually seeing the score for a specific product's implementation of Claude Opus 4.5. Transfer that model to a different harness and you get a different number.

The implication for builders is concrete: benchmark against your specific codebase and toolchain before committing to a framework. A tool that scores highest on SWE-bench may not score highest on your repo's particular mix of languages, test patterns, and architectural conventions. The scaffolding gap means leaderboard results are a ceiling, not a guarantee. And a separate paper this week (see Research) found that 150 instances of the same model analyzing the same financial dataset produced substantially divergent conclusions — reinforcing that model-level capability is necessary but not sufficient for reliable outcomes.