Issue Difficulty Is Predictable From Static Features at AUC 0.863, Driven by Patch Fragmentation and Repo Scale

arXiv 2608.18280·medium signalXBlueskyLinkedInCopy link

A large-scale study on CoderForge-Preview, described as the largest open dataset of coding agent trajectories to date (arXiv 2608.18280, 2026-08-18), extracts features from task patches, repositories and prompts and predicts agent success with AUC 0.863 using ensemble methods with SHAP attribution. The dominant drivers are patch fragmentation, meaning how many places a fix has to touch, and repository scale; prompt linguistic features only surface as top contributors in the mid-difficulty band, revealing a layered difficulty structure. That makes pre-hoc, static difficulty estimation possible and opens the door to difficulty-controlled benchmark construction instead of the current uninterpretable aggregate scores.