Agent Skills Work by Sequence, Not New Facts

A paper measuring 8,135 agent trials finds 65.7% of the benefit from Claude Code skills comes from procedural sequencing, not new facts, per an August 14 arXiv study.

That number changes what belongs in a skill file. Explicit knowledge injection accounts for just 4.5% of the benefit, so a skill written as background explanation wastes most of its tokens. A skill earns its keep by pinning down order: check this before that, run the test after the edit, never touch the lockfile directly.

The paper, "Demystifying Agent Skills: Why They Work; Until They Don't" (arXiv 2608.14036), comes from Zhiyuan Jiang and co-authors. The team open-coded 238 labels into a taxonomy of 12 skill-use modes and found skills beat Workflow Memory by 6.06 points in matched comparisons. The technique holds up. The assumed mechanism doesn't.

Retrieval breaks first

The second number is the one to act on. Expand a skill pool from 5 items to 100, and actual-use retrieval precision falls from 29.6% to 3.3%, the paper reports. That's about a ninefold drop, and it hits anyone who has spent the past year adding skills without pruning.

Claude Code 2.1.233 added claude plugin validate to check a .claude/skills directory for SKILL.md files with broken frontmatter, per the Claude Code changelog. It catches broken files, not the retrieval problem. The paper doesn't offer a fix for that side, and I haven't seen one shipped anywhere.

A skill library built past 20 or so entries most likely retrieves the wrong skill more often than the right one. Prune before adding.