BackendForge: Best Coding Agent (GPT-5.5) Deploys Correct Backends Only 28.6% of the Time Under a Strict Oracle

arXiv·high signal

BackendForge is a benchmark of 56 contract-defined backend-generation tasks rewritten from real open-source apps, where an agentic LLM must produce a Dockerized service judged only through black-box HTTP tests against an OpenAPI contract. A test agent and code agent co-evolve the oracle and reference implementation to harden evaluation without hidden requirements. The best model, GPT-5.5, succeeds on 55.4% of tasks under the base oracle but just 28.6% under the final oracle — concrete evidence that agents handle local API behaviors yet still can't reliably ship complete, deployable services.