A Shipped Plugin-Migration Agent Skill Adds 4.9 Points, but Regrading Exposes Judge Errors That Push the Interval Toward Zero

arXiv·low signalXBlueskyLinkedInCopy link

arXiv 2609.30120 re-examines 64 reports on 16 migration tasks for a coding-agent skill. The raw gain was 93.83 to 98.75, concentrated in one task. Hand review found grading errors in both directions, including a containment check that gave full credit when it accepted the parent directory. Corrected decisions moved the confidence interval to or across zero. Judges from two other model families agreed on 91.8% and 95.7% of decisions but estimated gains of 10.63 and 6.09 points, so skill evaluations depend heavily on which judge grades them.