Your agent skills only execute 56% of their own steps

I've written a lot of SKILL.md files. So has everyone reading this. The entire skills ecosystem is built on one assumption nobody measured: that when you write a procedure into a markdown file and load it into context, the model performs the procedure.

It doesn't. It performs about half of it.

University of Michigan researchers (Dantanarayana, Kashmira, Tang, Mars) measured 30 agent skills across two model generations and found that prose skill files, run through a normal tool-calling loop, cause the model to execute just 56% of the steps the skill itself mandates. The artifacts still pass output checks. The work looks done. Roughly half the mandated procedure never ran. (SIGIL, Zenodo)

Think about which steps get skipped. Not the ones that produce the visible artifact, because you'd notice. The verification steps. The "check that the migration is reversible" step. The "confirm no secrets in the diff" step. The steps whose absence looks exactly like success.

Their system, SIGIL, compiles a prose skill into an executable harness via AG-IR, a typed intermediate representation that separates model-owned cognition from code-owned mechanism. Step compliance goes to 86%. Full-procedure completion happens 2.3x as often. And it uses 0.58x the tokens, which is the part that should end the argument. Compilation isn't a reliability tax you pay. It's cheaper.

Here's the finding that kills the obvious rebuttal. The reflexive response to "models skip steps" is "so wait for a better model." Compiled compliance held flat at 86% across model generations while prose execution swung from 56% to 68%. Prose skills improve with model quality. They improve slowly, from a terrible baseline, and they don't converge. Skipped verification is a structural property of shoving procedures into context and hoping, not a capability gap that scales away.

Now hold that next to the ecosystem numbers from today's GitHub trending. affaan-m/ECC at 234,432 stars ships 281 on-demand skills (GitHub). addyosmani/agent-skills at 80,712 stars ships 23 lifecycle skills across six phases with explicit "verification gates" (GitHub). virgiliojr94/book-to-skill gained 989 stars in a day converting technical books into ~5,000-token skills (GitHub).

Every one of those is prose. Every one of those, by this measurement, runs about half its steps. ECC publishes no quantified benchmarks at all, which the researchers just gave us the vocabulary to notice: "281 skills" is a count of documents, not a count of procedures that execute.

What I'd do Monday: take the two or three skills you rely on where a skipped step actually costs you something, and pull the mechanical parts out of the prose into code the agent calls. Not the judgment. The mechanism. If a step is "run the tests and fail if any fail," that's a script, and a script runs 100% of the time. Leave the parts requiring taste in prose, because that's what the model is for.

The uncomfortable version: a skill file is a suggestion with confident formatting.