CivBench runs agents for 300+ turns over 76 MCP tools and finds they stop checking state before losing

arXiv·medium signalXBlueskyLinkedInCopy link

CivBench is an open-source long-horizon benchmark putting language model agents inside Civilization VI through MCP, with a single episode spanning over 300 turns and thousands of tool calls, 76 exposed MCP tools, and a narration layer converting visual game state to structured text. Across 23 admissible runs from four model families the authors decline to rank models and instead introduce two interface-level metrics: Proactive Monitoring Rate and RAG@10, whether commitments stated in planning reflections get executed within ten turns. Despite playbook instructions to query victory progress every 20 turns, agents queried every 30 to 75, and in 7 of 20 detectable defeats never queried inside the 20-turn warning window. Agents under-monitor state that is available but must be explicitly requested.