SpecHarness: agents claim completion 28.7 to 37.9 points more often than the official evaluator passes them

arXiv·medium signalXBlueskyLinkedInCopy link

On 509 source-grounded task requirements from SkillsBench, seven models satisfied only 79.6 to 86.4%, while their completion-claim rates exceeded official pass rates by 28.7 to 37.9 percentage points. SpecHarness compiles visible specs into source-linked obligations, and only admissible evidence from a qualified provider may mark one satisfied. The agent can request completion but cannot declare it. The result is another reason not to trust an agent's own 'done' signal in CI or ticket automation.