FDE-Bench: seven models solve 52.9 to 75.0% of 136 Docker, Compose and Kubernetes deployment tasks graded without an LLM judge

arXiv·medium signalXBlueskyLinkedInCopy link

arXiv 2609.27571 (23 Sept) scores agents on turning application code into running services across 136 greenfield and diagnose-and-repair tasks. Submissions are rebuilt and redeployed in a clean environment, then checked by 2,145 programmatic checks in four gated layers: build, readiness, behavior and spec conformance. A four-arm gate throws out any task that do-nothing or stub submissions can pass, and a vacuous health probe that passed readiness shows why the downstream checks are needed. The benchmark measures the deployment side of coding agents, which SWE-bench does not cover.