SIGIL Measures What Agent Skills Actually Execute: Prose Skills Run Only 56% of Their Own Mandated Steps; Compiling Them Into Typed Harnesses Gets 86% at 0.58x the Tokens
University of Michigan researchers (Dantanarayana, Kashmira, Tang, Mars) measured 30 agent skills across two model generations and found prose skill files — procedure documents loaded into context and run through a tool-calling loop — cause the model to perform just 56% of the steps the skill itself mandates, even when the resulting artifacts pass output checks. SIGIL compiles a prose skill into an executable harness via AG-IR, a typed intermediate representation separating model-owned cognition from code-owned mechanism, lifting step compliance to 86%, completing the full procedure 2.3x as often, and using 0.58x the tokens. Critically, compiled compliance held flat at 86% across model generations while prose execution swung from 56% to 68% — so skipped verification steps are a structural property of prose skills, not a model-quality problem you can wait out.
↳ Follow the thread