35 of 100 agent solutions that pass break when you change only the order the requirements arrived in
SpecPath (arXiv 2608.09799, Aug 10) holds repository, final contract, verifier, agent system, and execution budget constant and varies only the revision path by which a specification became final — the same end-state contract, reached through different histories of clarifications and amendments. Thirty-five of 100 successful implementations failed on at least one contract-equivalent history, and aggregate accuracy across paths looked nearly identical, which means path sensitivity is invisible to standard benchmarking. The practitioner takeaway: agents are partly following surface cues from how a request evolved rather than resolving requirements, so evaluate your agent against messy multi-turn requirement histories, not just the tidy consolidated ticket you'd write at the end.
↳ Follow the thread