Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-31 · high
It hit a 79.6% success rate building tasks from real pull requests, versus SWE-bench's flagged 13.6% misalignment rate.
Why now: The paper surfaced in the July 31 briefing, right alongside coverage of SWE-bench's own misalignment problem.
Change2Task converts a repository's merged pull requests into executable coding-agent tasks with working test environments, researchers report in a paper posted to arxiv.
The method verified successful task construction 79.6% of the time across five different task families, according to the paper. That matters for anyone grading a coding agent on work that resembles their own codebase, not someone else's repo, since eval quality shapes which agent a team ends up trusting with real work.
Most teams reach for SWE-bench for that grading. It has a flaw of its own: a 13.6% misalignment rate, per PAIChecker. The evidence doesn't specify what that misalignment measures, but the number alone is reason to look for an alternative.
Building an eval from your own merged PR history is that alternative. The paper doesn't say how the method performs on languages or frameworks outside its five families, or which repositories it was built and tested against. That makes 79.6% a starting point, not a guarantee.
Each link below shares sources, entities, or timing with this story.
The authors define goal-directed execution as four repeated behaviors: selecting goals, constructing task-relevant state, maintaining fidelity to higher-level objectives, verifying completion against the environment. Post-training Qwen3.5-122B-A10B on 363 long-horizon multi-to...
PAIChecker audits SWE-bench Verified and finds misalignment across five patterns and eleven scenarios, a direct consequence of the construction pipeline pairing a PR with whatever issue its description references, then using the issue as problem statement and the patch as test...
openJiuwen separates two harness problems it names Structural Composability and Runtime Adaptivity: developers compose capabilities across single agents, delegated sub-agents and a Swarm Flow over one shared execution layer, while the framework adapts context, feedback and tas...
arXiv 2607.28430 adds asynchronous message-passing to coding-agent harnesses with three primitives: threads, messages, and waiting for mentions, where the wait runs as a background task so an agent stays passively aware of teammates without blocking foreground work. Single Cla...
A paper from Xiao Yu, Baolin Peng, and Ruize Xu makes a claim that seems obvious once stated and is genuinely new as a training methodology: modern agents are inseparable from their inference harnesses, so training them in stripped-down RL sandboxes produces a train/serve mism...
While the capability stories pile up, here's the counterweight. As SWE-bench Verified scores cluster near saturation on July leaderboards, an enhanced analysis (SWE-Bench+, on the AIware 2026 benchmark track) found 60.83% of commonly resolved issues contain solution leakage ri...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.