Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-23 · high
An 8B version beat a 70B reward model and a 235B rubric generator at supervising the policy, per the paper.
Why now: DynamicRubric appears in a paper posted to arXiv on July 23, 2026, which says it already runs across WeChat Search's production traffic.
WeChat Search replaced its static reward model with an evaluator that rewrites its own grading rubric for every batch of responses, per a paper posted to arXiv (2607.20083). The new system, called DynamicRubric, now grades every response across WeChat Search's full production traffic, tens of millions of requests a day.
The problem is specific. As a policy model improves during reinforcement learning, its sampled responses start looking alike. A fixed evaluator's score gaps between them shrink toward zero, and with no gap there's no training signal.
The paper traces this to a probability-allocation argument. The gain from shifting probability mass between two candidate responses equals the evaluator's score gap between them. Collapse the gap and you collapse the gradient.
DynamicRubric's fix is to generate weighted binary rubric items conditioned on the candidate set being graded. That replaces a rubric written once and reused. No new loss function, no bigger model.
The size result is the part worth sitting with. An 8B DynamicRubric evaluator outperformed a 70B reward model and a 235B static rubric generator at supervising policy training, per the paper. That inverts the usual assumption that judge quality tracks judge size. Here it tracked whether the judge's rubric moved with the policy.
The paper doesn't say how often the rubric regenerates in production, or what the compute overhead looks like at that request volume. For teams running LLM-as-judge on a fixed prompt, that's a case for tying the judge's refresh cycle to the policy, not manual updates.
Each link below shares sources, entities, or timing with this story.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-18.
LLM uses OpenAI / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (LLM uses OpenAI); both cover LLM; earlier LLM coverage from 2026-06-19.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-14.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-22.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-10.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-10.