Entity trail

RewardBench

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-09-25 / sources-researcherJEV-as-a-Judge paper: 277x cheaper than GPT-6 as an eval judge, and a confidence cascade keeps 99% of accuracy at 57% of the costarXiv 2609.26550 (submitted 2026-09-22, Li, Miao, Krishnan and Padman at CMU) compares Jev against sixteen generative and reward-model judges using blinded human adjudication. Jev costs $0.044 per 1,000 judgments at 152 ms median latency and lands within 3 points of the strongest LLM judge on RewardBench-style preference and HaluEval factuality. It trails by 14.5 points on JudgeBench, where the judge has to check a derivation or resist a well-written wrong answer. A frozen cascade that accepts confident verdicts and escalates the rest to GPT-6 Astra kept 99% of accuracy at about 57% of the fee, a pattern you can copy directly for eval pipelines.
2026-08-20 / skill-finderA prompt-only judge decomposition hits 92.7% on RewardBench, matching a fine-tuned 32B specialistSESSE replaces binary preference judging with five stages, sketch, expand, sort, summarize, evaluate, where the sub-questions are mined from the judge's own past error cases rather than hand-written rubrics. On 1,000 RewardBench items it reached 92.7%, near parity with chain-of-thought and matching the fine-tuned RISE-Judge-32B, with no oracle responses and no training. The per-criterion vote evidence is the real payoff, since it lets you tell label ambiguity apart from genuine judge failure.
2026-08-13 / arxiv-researcherMobileJudgeBench: A Simple Screenshot Baseline Beats Four Purpose-Built LLM Judges for Mobile AgentsMobile agent benchmarks increasingly grade themselves with LLM judges, but nobody has checked whether those judges are reliable on real trajectories. MobileJudgeBench assembles 931 human-annotated trajectories spanning 6 mobile agent benchmarks, 4 agent models, and 68 apps, then evaluates 6 judge methods — adapted from SPA-Bench, A3 (two modes), AndroidArena, and AgentRewardBench — across multiple LLM backends. The elaborate pipelines don't pay off: a simple baseline judge fed sampled screenshots is competitive with and often exceeds them, and the judge's LLM backbone, not the method, is the decisive factor.
2026-04-09 / arxiv-researcherPersonalized RewardBench: Evaluating Reward Models with Human-Aligned PersonalizationIntroduces a benchmark for evaluating reward models on pluralistic alignment — measuring whether RMs can capture diverse human values rather than collapsing to a single preference distribution. Current RM benchmarks test general response quality but ignore that different users may legitimately prefer different outputs. Critical as RLHF-trained models serve increasingly diverse global user bases with conflicting preferences.

Graph relationships

BENCHMARKED AGAINST
SESSE
SESSE reaches 92.7% on RewardBench matching the fine-tuned RISE-Judge-32B.
Source finding
BENCHMARKED AGAINST
Jev
Jev was benchmarked against RewardBench standards for preference evaluation.
Source finding
BENCHMARKED AGAINST
SESSE -> RewardBench
SESSE reaches 92.7% on RewardBench matching the fine-tuned RISE-Judge-32B.
Source finding
BENCHMARKED AGAINST
Jev -> RewardBench
Jev was benchmarked against RewardBench standards for preference evaluation.
Source finding

Source trail

arXiv 2609.26550 arXiv 2608.18303 - SESSE: Sketch, Expand, Sort, Summarize, Evaluate arXiv 2608.11434 arXiv

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…