264K-Skill Study Detects When an Agent Skill's Description Lies About What It Actually Does

arXiv·high signal

As open-source Agent Skill marketplaces grow, users pick third-party skills from brief metadata that may not match true behavior — a gap the authors call cross-layer misalignment. Their PL-HCL (Progressive Loading-Aware Hierarchical Contrastive Learning) method models the layered structure of skills (metadata, instructions, execution resources) and learns cross-layer consistency, trained on a normalized corpus of over 264,000 skills plus a human-verified challenge set. It lifts Macro-F1 from ~0.45 for unadapted baselines to 0.87–0.89 across LLM backbones — a directly relevant defense for anyone installing skills from a marketplace.