deedchain benchmarks whether a browser agent's final report tells the truth about what it actually did
GitHub·medium signal
deedchain (created 4 September, 50 stars) argues that success-rate leaderboards only measure one quadrant of a 2x2 and are blind to hallucinated success and hallucinated failure. It extracts every concrete assertion from an agent's report, adjudicates each against recorded ground truth, and labels contradictions as OMISSION, FABRICATION, EXAGGERATION, or DISPLACEMENT. The whole pipeline is rule-based with no LLM judge, deliberately, because the authors treat judge-based evals as trivially gameable.