Claude Code adds an eval that scores skills against nothing

Claude Code 2.1.269 added a plugin eval command that runs each test three times with a skill loaded and three times without, according to Claude Code's plugin eval docs.

A case that scores 1.0 in both runs means the plugin added nothing to a task the model already handled alone. A skill that never fires still lets the model finish the job on its own. The absolute score reads as a pass, even though the delta sits near zero and the case's tool_used grader fails. Someone checking only the top-line score would call that a working skill.

There's a wrinkle in the scoring logic. Graders that check whether a skill was invoked can never pass in the without-plugin arm. Counting those as failures would drag the delta up artificially, so Claude Code marks them scored: false on both sides instead. Setting arm: both reverses that, useful for a must-not-invoke check with min: 0 and max: 0.

Six grader types come with the tool. Four run free (regex, tool_used, tool_order, file_exists) and two call a judge model (llm, baseline). CI gating runs through --threshold and --max-cost-usd flags. An untrusted checkout without --trust-plugin exits with an error when no terminal is attached, per MarkTechPost's coverage of the release. Granting Bash access routes a case through an OS sandbox that needs bubblewrap and socat on Linux, or WSL2 on Windows. Native Windows has no backend at all, so those cases just error out.

Run one eval against your most-used skill. If the delta comes back near zero, the skill isn't doing what you think it is.