One agent design beat baseline in 560 of 636 runs

A system called AgentX-Model ran 636 production recommender experiments and beat the business AUC baseline in 560 of them, per the AgentX-Model paper on arXiv. That's a production number, not a benchmark score. Recommender systems get graded on live traffic, not held-out test sets.

The design splits the work across two agents. A Research Agent writes proposals that get reviewed independently before anything runs. A Model Agent then executes multi-round experiments inside a sandbox, choosing its next move from four actions: Reproduce, Follow-up, Composition, and Diagnose.

Diagnose handles the failure mode that offline metrics miss. It's built to catch online feedback problems like PCOC prediction bias, the gap between what a model predicts and what converts once it's live. A model can look better in testing and still drift on a live signal. Catching that requires the loop to check production behavior, not just rerun the same offline comparison.

The loop structure is the real contribution. Each run's output determines the next question. The agent isn't working through a fixed list of experiments. It picks Reproduce, Follow-up, Composition, or Diagnose based on what the last run showed.

The paper doesn't say how these numbers hold up outside the production system it was tested against. It also doesn't say how much human review sat behind the "independently reviewed" proposal step. A 560-of-636 win rate is strong evidence from one deployment. Whether the four-action loop generalizes to other model classes is the open question.