LLM Memorization Is Surface-Form Dependent: Entity Name Formatting Changes Recall by Up to 40%

arXiv·medium signalXBlueskyLinkedInCopy link

Nishida et al. reveal that LLM factual recall is surprisingly sensitive to how entity names are written — not just whether the model 'knows' a fact. Changing surface forms (e.g., 'United States' vs 'US' vs 'USA') shifts recall accuracy by up to 40% on entity-based QA. This challenges the assumption that QA benchmarks measure stable factual knowledge and has direct implications for RAG pipeline design, where query reformulation could dramatically change retrieval-augmented accuracy.