Build internal agent benchmarks by reverse-engineering tasks from your own commits into colloquial role-played requests

arXiv 2607.20911·medium signalXBlueskyLinkedInCopy link

Tencent's WorkBuddy Bench (July 23) solves benchmark contamination without secrecy: each task is reverse-engineered from a real commit, PR, or business scenario and then rewritten as a short, colloquial, role-played request, so the task text is not web-searchable even though the whole suite is public — they rely on dataset versioning instead of a hidden test set. It spans repository-level engineering, front-end, office/business workflows, and red-/blue-team security, with different verification per subset and scores reported separately rather than averaged into one number. The construction method and the uniform task-directory format are the reusable parts for anyone building an internal eval from their own git history.