763 confirmed defects across 239 of 515 real agent skills, and the failures cluster at the boundary between declared intent and implementation

arXiv 2609.06052·high signalXBlueskyLinkedInCopy link

SkillSpec treats skill correctness as a Hoare-style specification problem: it turns a skill repository into a unified graph aligning descriptions, instructions and code, derives an ExpectSpec from declared intent, infers FactSpecs from encoded behavior under partially masked intent, then validates candidate defects in a sandbox. On 515 real skills from SkillsBench and widely downloaded repositories it flagged 763 manually confirmed defects across 239 skills at 61.2% precision. The node-level result matters most for anyone writing skills: specification reasoning is reliable on code nodes and breaks down on plain-text nodes, which is exactly where the free-form instruction prose lives.