Reused web-agent skills cut steps, until the library grows

Researchers behind EconSkills tested a different way for web agents to reuse experience. Instead of replaying full recorded trajectories, they distilled verified EconWebArena runs into parameterized procedures. Each one captures a task's scope, the navigation steps, site-specific guidance, verification checks, and recovery steps for when something goes wrong. Source-specific values get swapped for placeholders, so a skill built on one task can transfer to a similar one.

Anyone building an agent for repeat web tasks has to decide how it stores experience. Raw logs, or something more compressed.

In a controlled transfer test, matched skills beat prompting with no skill at all. They also used fewer steps on the tasks where both approaches succeeded. When the agent finds the right skill for the job, it finishes faster.

The catch shows up at library scale. Once the library grows large, retrieval has to pick among many candidates. Performance then drops back to about the same level as the no-skill baseline. Approximate matches on tasks the library doesn't cover offset the gains that correctly matched skills produce elsewhere.

The paper doesn't say how large a library has to get before the failure kicks in. It also doesn't specify what a coverage-aware retriever would look like in practice.