Kill position bias in LLM-as-judge: score both (A,B) and (B,A) and count only consistent wins

arXiv·medium signal

A systematic 2026 study of judge-bias mitigations confirms position bias is best handled by evaluating both orderings of a pair and only crediting a win when the verdict is consistent across both — and that self-preference bias requires using a different model family as the judge than the one being judged. For verbosity bias, switch to a 1–4 scale with an explicit reward for conciseness rather than open-ended scoring. These are cheap, deterministic harness changes that materially improve eval reliability.