Coloring Positive Words Green Shifts VLM Sentiment Predictions — a Styling-Only Prompt Injection Channel

arXiv 2608.14286·medium signalXBlueskyLinkedInCopy link

'Stealth Visual Prompts' change only the visual styling of text rendered as an image — color, contrast — while preserving semantics, and measurably steer vision-language model analysis. Coloring positive words green consistently pushes sentiment predictions positive to the point that VLMs fail to properly weigh negative words present in the same text, and the authors trace the effect to color-induced changes in the vision encoder's latent representations. Lowering text-background contrast increases reliance on visually salient cues and produces more wrong VQA answers. For anyone running VLMs over screenshots, resumes, or documents in a decision pipeline, this is an injection surface that survives any text-level sanitization. Code at github.com/KohsukeIde/color-bias-vlm.