Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-31 · high
It hit a 79.6% success rate building tasks from real pull requests, versus SWE-bench's flagged 13.6% misalignment rate.
Why now: The paper surfaced in the July 31 briefing, right alongside coverage of SWE-bench's own misalignment problem.
Change2Task converts a repository's merged pull requests into executable coding-agent tasks with working test environments, researchers report in a paper posted to arxiv.
The method verified successful task construction 79.6% of the time across five different task families, according to the paper. That matters for anyone grading a coding agent on work that resembles their own codebase, not someone else's repo, since eval quality shapes which agent a team ends up trusting with real work.
Most teams reach for SWE-bench for that grading. It has a flaw of its own: a 13.6% misalignment rate, per PAIChecker. The evidence doesn't specify what that misalignment measures, but the number alone is reason to look for an alternative.
Building an eval from your own merged PR history is that alternative. The paper doesn't say how the method performs on languages or frameworks outside its five families, or which repositories it was built and tested against. That makes 79.6% a starting point, not a guarantee.
Each link below shares sources, entities, or timing with this story.
Shared entities / Same source domain / Shared topic
Both cover PAIChecker, SWE; reported by the same outlet (arxiv.org); overlapping topics (construction, verified).
Shared entity: SWE / Same source domain / Shared topic
Both cover SWE; reported by the same outlet (arxiv.org); overlapping topics (codebase, coding agent, task).
Shared entity: SWE / Same source domain / Shared topic / Earlier coverage
Both cover SWE; reported by the same outlet (arxiv.org); overlapping topics (environment, task).
Both cover SWE; reported by the same outlet (arxiv.org); overlapping topics (eval, verified).
Shared entity: SWE / Same source domain / Earlier coverage / Tension
Both cover SWE; reported by the same outlet (arxiv.org); earlier SWE coverage from 2026-07-26.
Both cover SWE; reported by the same outlet (arxiv.org); earlier SWE coverage from 2026-07-23.
Shared entity: PRs / Same source domain / Earlier coverage / Tension
Both cover PRs; reported by the same outlet (arxiv.org); earlier PRs coverage from 2026-07-23.
Both cover PRs; reported by the same outlet (arxiv.org); earlier PRs coverage from 2026-07-21.