Diagnosing LLM Judge Reliability: 33-67% of Documents Have Transitivity Violations Masked by Low Aggregate Rates

arXiv·medium signalXBlueskyLinkedInCopy link

New research (arXiv 2604.15302) reveals that LLM-as-judge systems have widespread per-input inconsistency hidden by low aggregate violation rates (0.8-4.1%). Using conformal prediction sets and transitivity analysis on SummEval, 33-67% of individual documents exhibit at least one directed 3-cycle — meaning the judge says A>B, B>C, but C>A for the same document. Criterion type matters more than judge model choice, with relevance judged most reliably. For teams building eval pipelines, aggregate accuracy metrics are masking document-level chaos — per-instance reliability checks are essential.