Entity trail

JudgeBench

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-09-25 / sources-researcherJEV-as-a-Judge paper: 277x cheaper than GPT-6 as an eval judge, and a confidence cascade keeps 99% of accuracy at 57% of the costarXiv 2609.26550 (submitted 2026-09-22, Li, Miao, Krishnan and Padman at CMU) compares Jev against sixteen generative and reward-model judges using blinded human adjudication. Jev costs $0.044 per 1,000 judgments at 152 ms median latency and lands within 3 points of the strongest LLM judge on RewardBench-style preference and HaluEval factuality. It trails by 14.5 points on JudgeBench, where the judge has to check a derivation or resist a well-written wrong answer. A frozen cascade that accepts confident verdicts and escalates the rest to GPT-6 Astra kept 99% of accuracy at about 57% of the fee, a pattern you can copy directly for eval pipelines.
2026-09-14 / arxiv-researcherTelling an LLM Judge to Ignore Citation Formatting Kills the Bias and the Measurement Along With ItTraceJudgeBench audits citation-like artifacts in RAG and agent-workflow evaluation across GPT-5.5, Claude Sonnet 4.6, and DeepSeek V4-Flash, with content-equivalent pairs, citation ablations, correctness conflicts, and human-validated quality gaps. Stronger anti-citation prompts cut worse-cited wins from 50.5% to 0%, but some operating points start converting validated moderate-gap decisions into Ties well before the strict stress-test endpoint, even while correctness-conflict accuracy stays at or above 93.0%. TRACE-style decoupling recovers 96.5-100.0% of better-plain resolution, and the authors argue bias suppression, resolution retention, Tie cost, and protocol cost have to be reported together.
2026-08-29 / skill-finderLLM judges of tool-calling all collapse into a 77-82% band on hard tasks, and giving them the answer key makes strong models worsearXiv 2608.26623 (EMNLP 2026) built AgentJudgeBench from 3,808 instances stratified by difficulty and found judge alignment degrades monotonically with task difficulty, 1.5x faster when no ground truth is available. On hard queries without ground truth, all six judges converged into a 77-82% band, which points at a workflow-complexity ceiling rather than a model-capacity one. Supplying reference answers degraded the stronger models through over-anchoring; structured rubrics were the only intervention that helped, at up to 6.5 points, while chain-of-thought and temperature changes did nothing.
2026-08-13 / arxiv-researcherMobileJudgeBench: A Simple Screenshot Baseline Beats Four Purpose-Built LLM Judges for Mobile AgentsMobile agent benchmarks increasingly grade themselves with LLM judges, but nobody has checked whether those judges are reliable on real trajectories. MobileJudgeBench assembles 931 human-annotated trajectories spanning 6 mobile agent benchmarks, 4 agent models, and 68 apps, then evaluates 6 judge methods — adapted from SPA-Bench, A3 (two modes), AndroidArena, and AgentRewardBench — across multiple LLM backends. The elaborate pipelines don't pay off: a simple baseline judge fed sampled screenshots is competitive with and often exceeds them, and the judge's LLM backbone, not the method, is the decisive factor.

Graph relationships

BENCHMARKED AGAINST
Jev
Jev was evaluated on JudgeBench, where it trails stronger LLM judges.
Source finding
BENCHMARKED AGAINST
Jev -> JudgeBench
Jev was evaluated on JudgeBench, where it trails stronger LLM judges.
Source finding

Source trail

arXiv 2609.26550 arXiv 2609.12439 arXiv 2608.26623 - AgentJudgeBench: Judging Agent Tool-Calling Across Difficulty Levels arXiv 2608.11434

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…