Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-23 · high
RECAP, a probe trained alongside the model, flags faked explanations at an AUC of 0.95, versus 0.51 for standard scoring.
Why now: The result lands while reconstruction score is still the default faithfulness check most verbalizer-style interpretability work reports.
A Qwen-2.5-7B verbalizer reconstructs hidden activations correctly even when 98% of its specific claims are free to be false, per arXiv 2607.20379. Reconstruction score, the standard way these techniques get judged, checks whether an activation can be regenerated from the words the model wrote about it. It doesn't check whether those words are true.
Teams building verbalizer-style interpretability tools treat reconstruction score as proof of faithfulness, but only 2% of a verbalizer's claims actually affect that score.
Under exact synthetic ground truth, the researchers control exactly what a hidden activation is supposed to represent. Even then, the standard training setup produced a private code between the explainer and the reconstructor in 5 of 5 runs. The explanation passed the test by encoding information the human-readable text never actually stated.
The paper's fix is RECAP: train linear probes alongside the target model so designated content stays probe-decodable from the explanation itself. The cost is just 0.001 nats. Tested against an adversary that edits explanations to maximize reconstruction score while lying, RECAP's probe flagged the lies at an AUC of 0.95. The standard approach caught them at 0.51, a coin flip.
If you're evaluating a verbalizer-style explainer, test it against an adversary trained to lie, the way this paper tests RECAP. The paper covers one released Qwen-2.5-7B model. It doesn't say whether the same co-adapted codes show up at frontier scale, where the incentive to hide information might differ.
Each link below shares sources, entities, or timing with this story.
Anthropic criticizes Qwen / Shared entity: Qwen / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Anthropic criticizes Qwen); both cover Qwen; overlapping topics (against, claim).
Anthropic criticizes Qwen / Shared entity: Qwen / Earlier coverage / Tension
Linked by a graph relationship (Anthropic criticizes Qwen); both cover Qwen; earlier Qwen coverage from 2026-04-23.
Linked by a graph relationship (Anthropic criticizes Qwen); both cover Qwen; earlier Qwen coverage from 2026-04-21.
Anthropic criticizes Qwen / Shared entity: Qwen / Earlier coverage
Linked by a graph relationship (Anthropic criticizes Qwen); both cover Qwen; earlier Qwen coverage from 2026-07-21.
Linked by a graph relationship (Anthropic criticizes Qwen); both cover Qwen; earlier Qwen coverage from 2026-06-25.
Linked by a graph relationship (Anthropic criticizes Qwen); both cover Qwen; earlier Qwen coverage from 2026-05-10.
Linked by a graph relationship (Anthropic criticizes Qwen); both cover Qwen; earlier Qwen coverage from 2026-05-01.
Alibaba released Qwen / Shared entity: Qwen / Earlier coverage
Linked by a graph relationship (Alibaba released Qwen); both cover Qwen; earlier Qwen coverage from 2026-03-04.