DeepMind's XYEval: one confident, misleading user hint cuts agent scores by up to 46.7%

arXiv / Google DeepMind·medium signalXBlueskyLinkedInCopy link

XYEval (arXiv 2609.23939, submitted 2026-09-20; authors include Zhengxuan Wu, Oyvind Tafjord and Been Kim) turns existing benchmarks into 'XY problem' tests by injecting a plausible but wrong user suggestion. Across five models and six suites, relative drops reach 46.7%, and agents often disagree with the hint in their reasoning and then follow it anyway. On tau2-bench, a pedantic user who demands explanations before approving a better plan causes even larger drops. A system prompt that warns about XY problems only partly helps, so agent harnesses need an explicit verify-then-push-back step. Code is at github.com/google-deepmind/xyeval.