Run a dedicated spec-elicitation agent before the coding agent — +6.9–21.3% pass rate on a benchmark where frontier models score under 1%

arXiv 2607.27167·medium signalXBlueskyLinkedInCopy link

ProgramBench gives an agent only natural-language docs plus an execute-only binary as behavioral oracle, and frontier models solve under 1% of its 200 instances — building from scratch is where current harnesses collapse. SpecFirst (July 29, 2026) splits the single loop in two: a spec agent probes the binary and fuses observations with documentation into a structured specification, then a separate synthesis agent codes against that fixed reference. Test pass rates rose 6.9–21.3% and binary exploration coverage 9.4–18.5% across four models spanning an order of magnitude of capability, all statistically significant — the gain comes from resolving documentation ambiguity before any code exists rather than letting early misreadings propagate.