Harden your LLM-as-judge mechanically: shuffle, pin, rotate families, calibrate — and apply IRT to the judges

Future AGI·medium signal

The 2026 consensus names five judge biases (position, verbosity, self-preference, format, calibration drift) and the mitigations that actually survive production are mechanical: shuffle order on every pairwise call and only count consistent wins, pin the judge contract/version, rotate judges across model families (especially to defeat self-preference), and calibrate against humans. A newer line applies Item Response Theory to the judges themselves — treating reliability as a property of the measurement instrument and flagging rubric criteria that are too ambiguous or too sensitive. Builders running eval pipelines should bake order-swapping and cross-family rotation into the harness rather than trusting a single judge model's score.