Equal AI evaluator scores can hide opposite behavior

A paper released on arXiv tested five evaluator types, lexical, character-level, semantic, LLM-based and hybrid, against four correctness checks and found that none of them passed all four. That matters for anyone wiring an LLM judge into a production eval setup. Two evaluators that score nearly the same on aggregate benchmarks can behave in opposite ways on the same input.

The researchers built the checks around controlled response transformations, edits to a model's output with a known, expected effect on its score. They ran the five evaluator types through four tests: stability, sensitivity, repeat-run variability and reproducibility. No evaluator satisfied all four, and the paper doesn't say which comes closest.

The sharper result is that evaluators with near-identical aggregate performance had substantially different behavioral profiles. A judge that matches a competitor's overall accuracy score can still diverge on how it treats individual edge cases. It can also give a different verdict on the same input from one run to the next.