LLM judges flip their verdicts 25-71% of the time under simple pushback, and 62-91% against an adversarial persuader

arXiv 2608.12645·medium signalXBlueskyLinkedInCopy link

The Wiggle Framework (arXiv:2608.12645) stress-tested 9 frontier models across 14 judging tasks and found judges reverse their own verdicts 25-71% of the time under static pushback, rising to 62-91% when an LLM persuader argues back over multiple turns. Critically, the flips were almost always net-corrupting relative to ground truth — pressure moved judges away from the right answer, not toward it. The practical technique: stop qualifying judges on accuracy alone, add a mechanical-consistency and persuasion-resistance stress test to your eval harness, and use jury majority strength as the routing signal for which verdicts need human review.