When a model gets a commonsense question wrong, re-ask it stripped of task framing — a context-free probe recovers over 90% of the failures

arXiv 2607.28478·medium signalXBlueskyLinkedInCopy link

The SaliTrap benchmark isolates 'salience bias': explicit but useless distractors in a prompt (numeric values in particular) hijack the model into ignoring the implicit physical or commonsense prerequisites of the task, and severity scales with distractor density. Across 12 state-of-the-art models, detecting the trap turned out to be largely decoupled from actually avoiding it. The diagnostic that matters for builders: re-eliciting the same model with the task framing removed recovers over 90% of sycophantic-compliance failures, proving the knowledge is present but suppressed — and lightweight inference-time prompting alone substantially closes the gap with no retraining.