NVIDIA Finds Structural Scans of Agent Skill Files Correlate With Actual Usefulness at Spearman 0.14, Proposes Measuring Skill Lift Instead
Latent Space (AINews), reporting NVIDIA evaluation work via @omarsar0·high signal
New NVIDIA evaluation work, summarized in AINews on 2026-08-25, tested whether linting an agent skill file for structure predicts whether the skill helps, and got a rank correlation of just 0.14 against judged quality. The proposed replacement is Skill Lift: run the identical task twice under identical conditions, once with the skill loaded and once without, and score the delta in completed work. Anyone maintaining a directory of SKILL.md files is currently grading them on the wrong axis.