MaliciousSkillBench: Skill Scanners Hold 95.6% Recall but Flag 62.4% of Benign Skills From Unseen Sources

arXiv 2608.19901·high signalXBlueskyLinkedInCopy link

arXiv 2608.19901 consolidates 13 public sources into MaliciousSkillBench, reducing 8,414 raw malicious records to 7,539 normalized-unique identities across 4,588 structural families, yielding a primary benchmark of 9,740 Skills (7,505 malicious, 2,235 benign) with 11 harmonized attack categories. Learned text detectors score 0.882-0.932 Macro-F1 under random splits but collapse to 0.653-0.665 when evaluated source-disjoint; the strongest word TF-IDF SVM keeps 95.6% malicious recall while producing a 62.4% false-positive rate on benign skills from held-out sources. Three off-the-shelf skill scanners cut false positives only by giving up most of their malicious recall, so anyone gating a skill marketplace today is choosing between alert fatigue and blind spots.