When Tool Output Contradicts the Prompt, LLMs Follow Recency Over Stated Source Reliability

arXiv 2608.20116·medium signalXBlueskyLinkedInCopy link

arXiv 2608.20116 builds a synthetic benchmark where latent risk trajectories generate both numerical time series and natural-language summaries, letting the authors construct conflicts in which exactly one evidence source matches ground truth while independently varying modality, recency, stated source reliability, and provenance. Across open-weight instruction-tuned models the arbitration is systematic rather than noisy: models hold distinct text-versus-number preferences, weight temporal recency more consistently than explicit reliability cues, and over-trust external forecasts even when those contradict direct evidence already in context. For anyone wiring tools into a decision system, that means an out-of-date but recent-looking tool response can quietly outrank a correct fact in the prompt.