Adding tools and skills to a working agent makes it fail tasks it used to pass

EVOHARNESSBENCH does something I haven't seen a benchmark do: it holds the task stream fixed and evolves the harness (arXiv 2609.04280). Seventeen multi-stage streams built from 802 tasks, 520 tools, 42 skills and 62 agents. The finding is that harness expansion alone degrades performance on tasks the agent previously solved. The authors call it harness-induced forgetting. Retention and adaptation pull in opposite directions, so an agent configured to hold onto old competence absorbs new capabilities worse, and vice versa.

Read that against how everybody actually works. You have an agent that handles your codebase. You add an MCP server for your database. You add three skills you found in a catalog. You add a browser tool. At no point do you re-run the tasks that worked in week one, because they worked. This paper is the first controlled evidence that the additions are themselves a regression source, independent of any bug in the thing you added.

The mechanism isn't mysterious once you say it out loud. Every tool description, every skill frontmatter, every server capability list occupies context and competes for the model's attention at selection time. A model choosing among nine tools and a model choosing among sixty are doing different tasks. Claude Code 2.1.261 added /skill-doctor on September 4, which prices every loaded skill and names the ones never invoked, and that tool exists because someone at Anthropic hit this wall.

There's a related result from an OSWorld study that keeps me from reading EVOHARNESSBENCH as "don't add anything." An evolving skill library beat a configuration-matched empty-library control by 5.7 to 18.6 points across four application domains with the action-generation and grounding stacks held identical (arXiv 2609.04869). Skills help. But the same paper's provenance analysis found revision churn in GIMP where repeated accepted edits failed to recover the task the skill came from. Gains are real and conditional, not monotone.

So the practice is boring and nobody does it: keep a regression suite of tasks your agent already passes, and re-run it after every harness change. Not the new capability. The old ones. If you can't afford to run the whole suite, run five tasks. Five is infinitely more than zero, which is what most of us run today, including me until about an hour ago.

The Notion story and this one are the same argument from opposite ends. What you bolt onto the harness is now both your risk surface and your regression surface.