MCPGen: models reconstruct MCP workflows at 88.5% but no model exceeds 57% end-to-end execution
MCPGen is an executable benchmark of 100 self-contained MCP projects across 16 domains, scoring workflow reconstruction, tool creation and backward-compatible workflow extension through static analysis, unit and integration tests, and process-isolated end-to-end runs rather than trajectory proxies. Across 11 LLMs the gap is where builders will feel it: structural plausibility is nearly solved at 88.5% reconstruction, per-tool unit tests reach 63.8%, but project-level integration tops out at 45% and end-to-end execution at 57%. The cross-layer consistency between graph structure, tool implementations, schema bindings and runtime wiring is the part that breaks, which means generated MCP servers need integration tests, not unit tests, to catch what is actually wrong.
↳ Follow the thread