ASMI Measures Uncertainty by Masking Attention Heads — Halving Retained Error on Confident-but-Fragile Predictions

arXiv 2608.11138·medium signalXBlueskyLinkedInCopy link

The paper argues token uncertainty shows up not only in output-distribution breadth but in whether a confident prediction is fragile under perturbation of its attention pathways. ASMI (Attention-Subnetwork Mutual Information) is training-free: it masks attention heads and measures BALD mutual information among the resulting subnetworks through a semantic-agreement kernel. On grounded QA it adds error-predictive information beyond single-pass confidence and entropy, concentrated exactly in confident-but-fragile predictions where acting on it roughly halves the retained error of a confidence filter; the single-response variant ties or beats Semantic Entropy on 10 of 12 grounded benchmark-backbone settings. Crucially, on parametric QA all variants revert to or below the zero-cost MSP baseline — so this helps for retrieval-grounded answers, not closed-book ones.