Agent Skills Work by Habit, Not Knowledge

Agent skills work mainly by locking in a stable execution path, not by teaching new facts, per an arXiv study of 8,135 trials posted August 14, 2026. Procedural anchoring accounts for 65.7% of the measured benefit, versus 4.5% for explicit knowledge injection. That's the opposite of what most SKILL.md files are written to do.

The paper is titled "Demystifying Agent Skills: Why They Work, Until They Don't," by Zhiyuan Jiang and coauthors. It normalized 8,135 trial records into 238 labels across 12 skill-use modes.

In matched comparisons, per the paper, skills beat Workflow Memory by 6.06 points. That backs the anchoring finding: a fixed procedure beats a stored memory of past runs.

The failure mode sits in retrieval, not content. Growing the skill pool from 5 to 100 items drops actual-use precision from 29.6% to 3.3%, per the paper's trial data. A near-collapse, not a gradual decline, and it hits anyone stockpiling skills expecting the agent to find the right one.

I've been building with Claude Code in my personal projects long enough to collect a pile of half-used skill files myself. This gives me a number to point at instead of a hunch. If your agent has dozens of skills sitting around and results have gotten worse, check retrieval first, not the content.

Skill libraries that grow past a few dozen items will fail on retrieval before they fail on content. Most teams are optimizing the wrong 4.5% of the benefit. Watch whether builders respond by capping skill pools or by building retrieval that works at scale. The paper's own numbers argue for the cap: a small, curated skill set beats a sprawling library an agent can't find.