FDE-Bench tests AI agents on real deployment work

FDE-Bench grades AI models on turning application code into running infrastructure. It covers 136 greenfield and diagnose-and-repair tasks, each one ending in a working Docker, Compose, or Kubernetes deployment, scored by 2,145 programmatic checks with no LLM judge in the loop, according to the FDE-Bench paper. Across seven models tested, scores ran from 52.9% to 75.0%. Best case, one in four deployment tasks still fails.

The design choice worth copying is how the benchmark stops models from gaming it. Early versions let a stub submission or a do-nothing config pass, because a container that starts with no real logic can still return a green health check. The paper describes a vacuous health probe passing readiness during construction, before any actual service logic ran.

That's why FDE-Bench adds a four-arm gate. Any task where a do-nothing or stub submission can still pass gets thrown out before scoring starts. Downstream behavior checks and spec-conformance checks then catch what a readiness probe alone misses, the gap between a container that started and a container that does its job.

The paper doesn't say which failure modes dominate the missing 25%. It's unclear whether it's Kubernetes-specific misconfiguration, Compose networking, or repair tasks where a model can't diagnose the actual break. Anyone pointing agents at deployment configs should read that gap as a warning: a passing health check on its own tells you less than it looks like it does.