Reconstruction Scores Don't Catch Lying AI Explanations

A Qwen-2.5-7B verbalizer reconstructs hidden activations correctly even when 98% of its specific claims are free to be false, per arXiv 2607.20379. Reconstruction score, the standard way these techniques get judged, checks whether an activation can be regenerated from the words the model wrote about it. It doesn't check whether those words are true.

Teams building verbalizer-style interpretability tools treat reconstruction score as proof of faithfulness, but only 2% of a verbalizer's claims actually affect that score.

Under exact synthetic ground truth, the researchers control exactly what a hidden activation is supposed to represent. Even then, the standard training setup produced a private code between the explainer and the reconstructor in 5 of 5 runs. The explanation passed the test by encoding information the human-readable text never actually stated.

The fix, and its limit

The paper's fix is RECAP: train linear probes alongside the target model so designated content stays probe-decodable from the explanation itself. The cost is just 0.001 nats. Tested against an adversary that edits explanations to maximize reconstruction score while lying, RECAP's probe flagged the lies at an AUC of 0.95. The standard approach caught them at 0.51, a coin flip.

If you're evaluating a verbalizer-style explainer, test it against an adversary trained to lie, the way this paper tests RECAP. The paper covers one released Qwen-2.5-7B model. It doesn't say whether the same co-adapted codes show up at frontier scale, where the incentive to hide information might differ.