Four RLVR verifiers disagree by up to 41 points

Four RLVR verifiers disagreed by up to 41 points on identical, meaning-preserving answer rewrites, per a new metamorphic-testing paper. Any RL training run that scores outputs with one of these verifiers inherits that gap as reward noise before a model gets judged on the merits.

The method targets the verifier, not the model. Researchers generated certified rewrites that keep an answer's meaning fixed while changing its form, so any rejection counts as a provable false negative. They ran 307,420 verdicts through the test.

Self-validation, a verifier checking a rewrite of its own accepted answer, ranged from 53.8% to 95.2% depending on which verifier ran. That's a 41.4-point gap on inputs that should score the same every time.

Two configurations of one library disagreed with each other on 49.9% of pairs. The settings mattered almost as much as a coin flip.

Whitespace and punctuation alone caused 93.0% of the failures in the default LaTeX config, answers marked wrong for formatting, not substance.

One verifier's numeric cascade showed a sharper failure mode. It rejected every off-by-one wrong answer under 10,000, then accepted every off-by-one wrong answer at 10,000 or above. Being wrong got rewarded once the number was large enough.

The paper doesn't say whether teams using these verifiers have audited their own false-negative rates.