Mining 1,000 repos into 5,000 skills lifted MLE-bench 134% with the model frozen

Repo-To-Skill (arXiv 2609.02749, submitted September 2) makes the argument I've been circling for months and then supplies the controlled experiment.

The claim is that what research agents lack is operational know-how, not reasoning capability, and that this know-how already exists, written down, inside the GitHub repositories those agents depend on. So they extracted it: 5,000+ verified skills from 1,000 widely used ML repos, organized into 20 areas and 178 capability families, forming what they call the AREX-Skill Library.

The controls are why this is credible. GPT-5.5 backbone held constant. Harness held constant. Execution budget held constant. The only change is the skill library. MLE-bench up 134.3%. PaperBench up 34.4%. FrontierCS up 9.2%. PassNet up 14.0%.

Nothing about model choice. Nothing about scaffolding cleverness. Just: give the agent the procedural knowledge that was already sitting in the repos it's calling.

Three other papers from the same week point the same way. SkillGLoW attacks the storage side, arguing a single global skill document collapses into generic advice while a flat per-task pool inflates and stays welded to the instance that produced it. Its fix is to consolidate skills by procedural family and de-instantiate them into priors, which gained 17.2 points on hard tasks with positive gains in all 12 continual-improvement runs, kept one prior per family (3.6x more compact than per-task storage), and lifted unseen ALFWorld success from 73.9% to 83.9%. MASkills does credit assignment over skill libraries for multi-agent systems. CHIME splits memory into a planning bank and an execution bank and attributes blame to one before writing to either, on the argument that outcome-only memory conflates plan quality with execution error.

The practical version for a working engineer, today: your project's dependencies contain documented operational knowledge that your coding agent rediscovers from scratch every session. Pick your three heaviest dependencies. Read their docs, their examples directory, and their issue templates. Write one skill file per capability family, not per task. Keep them de-instantiated, meaning the skill describes the procedure rather than the specific call you made last Tuesday.

I've done a version of this by hand and the payoff is real but slower than 134% suggests, because my dependency set is smaller and my tasks are more varied than MLE-bench. What the paper convinced me of is the direction: spending an afternoon writing skills beats spending the same afternoon evaluating a model swap. The model swap is the more exciting decision and it's usually the worse one.

The caveat that would make me happier: 5,000 verified skills is a lot of extraction, and the paper doesn't make it obvious how much of the gain survives at 50 skills, which is what a solo builder can realistically produce.