Policy Invariance Test Exposes Fragility of LLM Safety Judges — Verdicts Depend on Wording, Not Behavior

arXiv·medium signal

LLM-as-a-Judge safety pipelines are shown to fail a basic reliability requirement: policy invariance. Verdicts change based on how the evaluation rubric is worded rather than what the agent actually did. The paper operationalizes three testable principles: rubric-semantics invariance, rubric-threshold invariance, and ambiguity-aware calibration. Critical for anyone relying on LLM judges for safety evaluation in production agent systems.