Adding tools and skills to an agent harness makes it fail tasks it used to pass, and there is now a benchmark measuring it

arXiv 2609.04280·high signalXBlueskyLinkedInCopy link

EVOHARNESSBENCH holds the task stream fixed and evolves the harness instead, across 17 multi-stage streams built from 802 tasks, 520 tools, 42 skills and 62 agents. Harness expansion alone degrades performance on previously solved tasks, which the authors name harness-induced forgetting, and retention and adaptation pull in opposite directions so preserving old competence does not help absorb new capabilities. If you keep bolting skills and MCP servers onto a working agent, this is the first controlled evidence that the additions themselves are a regression source, and it argues for re-running an old task suite after every harness change.