Sparse Autoencoder Comparisons Are Measuring Token Position, Not Latents — and the Fix Is One Line

arXiv 2608.13337·medium signalXBlueskyLinkedInCopy link

The standard SAE evaluation protocol ablates a latent and measures the effect at the token where it fires hardest, but that token is chosen by the dictionary under evaluation, so two dictionaries get compared at different places. Matching latents by decoder similarity across two Google-released SAEs for the same model, the authors found the pair picks different measurement tokens for a large share of near-identically-encoded latents; training six autoencoders from one initialization showed 7.6% and 11.9% of apparent inter-dictionary variance collapses to near zero once position is held fixed. Worse, across a sixteenfold range of corpus sizes dictionaries agreed *less* about where to measure, so the problem grows with scale — the paper audits five published papers against a corrected protocol.