Treat your LLM judge as unreliable until calibrated: 8 June studies show coin-flip self-agreement

NextFuture·high signal

Eight studies published June 13–17, 2026 found LLM judges disagree with themselves at near coin-flip rates on repeated identical-prompt runs, score gaps swing with inference budget alone, and most eval tools make it easy to run a judge but hard to prove it agrees with humans — one paper literally titled 'The Coin Flip Judge?' after 50× repeated pairwise/pointwise runs. The fix is a real pipeline: a judge-prompt registry, a calibration job against an expert-labeled gold set, and a drift monitor that alerts on Cohen's-kappa drops. The blunt takeaway for builders: an under-validated judge is worse than none — it manufactures false confidence at scale.