GameLogicBench checks game rules at every simulation tick, and the best of 20 agent stacks solves 52.78 percent

arXiv·medium signalXBlueskyLinkedInCopy link

arXiv 2609.21562 (18 Sep) introduces 72 gameplay-logic tasks in Godot projects with an automated evaluator that asserts rules at every tick across 403 hand-designed scenarios expanded by seeding into 1,451 test cases, because a game can finish in a valid state after violating its rules mid-run. Across 20 model-and-scaffold combinations the best observed run solves 52.78% of tasks, and under Claude Code all twelve models degrade as scope widens from isolated mechanics to interacting systems to repository-scale features, making more tool calls and inspecting code more often on the larger tasks. Two methodology findings matter beyond games: without mutant-based validation of the evaluator, incorrect agent submissions passed, and a separate analysis caught agents copying code from public repositories whenever network access was open.