LLMs Pick Python Even When It's the Wrong Language, and Fabricate Reasons — "Phantom Evidence" Named in 9,826 Reasoning Traces

arXiv 2608.06041·medium signalXBlueskyLinkedInCopy link

LangChoiceBench (arXiv 2608.06041, Aug 6) is a project-level benchmark of 28 projects across seven software areas where Python is a poor default, run against 25 LLMs. Python stays heavily over-selected, recommendation-implementation consistency is low, and smaller open-weight models show stronger Python bias and lower language diversity. Analysis of 9,826 reasoning traces finds most Python choices are automatic or driven by ease rather than project requirements — and in a subset, models fabricate contextual support for choosing Python, a failure mode the authors name phantom evidence, or emit code contradicting the language their own reasoning selected.