GPT-5.5 ships a working backend 28.6% of the time

GPT-5.5 solves 55.4% of BackendForge's 56 backend tasks under the benchmark's base oracle. Switch to the hardened oracle and that score drops to 28.6%, per the arXiv paper. Same model, same tasks, same agent. The only thing that changed is how hard the tests look.

BackendForge judges agents through black-box HTTP requests against an OpenAPI contract, not by reading the code. There's no partial credit for an endpoint that mostly works. Either the service responds correctly to the contract or it doesn't.

That 27-point gap between the two oracles is the real finding. Agents are good at local API behavior and bad at shipping a complete, deployable service. An endpoint can return the right JSON in the happy path. It can still fail the moment a hardened test hits an auth edge case, or checks whether a migration actually ran.

I see this building solo. An agent writes the route and a test for that route. The test passes, so I move on. Later I find the migration never got wired up, or the auth check only covers the one case the agent thought to test. The tests it wrote pass. The service doesn't ship.

What to watch

Watch how any benchmark defines success before trusting a headline pass rate. A number like 55.4% sounds like an agent is most of the way to reliable backend work. The hardened number, 28.6%, is closer to what happens once you actually deploy the thing. If you're grading your own agent's output, grade it the hard way. Use black-box HTTP tests against the real contract, not the tests the agent chose to write for itself.