Stress-test your LLM judges with the RAND Judge Reliability Harness before relying on them

arXiv·medium signal

RAND's open-source Judge Reliability Harness (JRH, March 2026) stress-tests LLM judges across consistency and discriminative tests — and found judge consistency breaks down on inputs as trivial as formatting changes, paraphrasing, and verbosity shifts. Before wiring an LLM judge into CI or launch gates, run it through JRH to quantify where it's unreliable, rather than assuming a frontier model judges stably.