Vibe Coding
FrontierHarness Eval: same model, nine harnesses, 17.5x cost spread and a 16-point pass-rate spread
The frontier-harness-eval/eval repo (144 stars, created 2026-08-31, pushed 2026-09-06) ran Kimi K3 through nine coding-agent harnesses in 12 configurations over the same 30 software-engineering tasks, 360 runs total. Codex led on pass rate at 66.7% ($3.47 median cost per pass), Claude Code matched DSH Creator at 63.3% but cost $18.34 per pass with only 67.8% median cache hit, and Exo Harness passed 53.3% at $1.05. The harness, not the model, is the variable nobody controls for, and cache behaviour is where the 17.5x cost gap lives.
Source
↳ Follow the thread