CordisBench measures whether a model can reason about the harness it is allowed to rewrite

arXiv·medium signalXBlueskyLinkedInCopy link

Dynamic agent harnesses let a model change the software shaping its own execution, and a local plugin change can propagate through dependencies and cleanup. CordisBench is a 1,200-question benchmark pairing a controlled formal setting with programs executed against Cordis, a runtime managing component dependencies and teardown, asking models to identify affected components, predict state after a given teardown order, determine which conditions hold under all or some orders, and pick reconfigurations that execute successfully. Models handle small systems but degrade as relevant interactions grow, especially on final state and cross-teardown-order reasoning, and the cost is real: on the 16-interaction subset GPT-5.6 Luna burns nearly 3,000 reasoning tokens per question at medium effort.