Every LLM judge grades stronger models more leniently, and you can correct it without labels.

Across four benchmarks and 36 judge-examinee pairs, a model's own task accuracy predicts its judging accuracy at r ≥ 0.90, but capability doesn't buy fairness: more capable examinees get more lenient judgments from every judge at r ≥ 0.83. Calibrated weighted majority voting estimates each judge's error rates purely from inter-judge disagreement, needing no ground truth, and stays within 0.5pp of an oracle under distribution shift.