Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-09-10 · high
Researchers got GUI agents to run a planted terminal command and then finish the user's original task with no sign onscreen that anything else happened.
Why now: Covered in the September 10 research roundup.
A trained image patch, planted on a web page, got computer-use AI agents to execute a malicious terminal command and then carry on with the user's original task like nothing happened.
That second part is the problem. Researchers behind AgentHijack deployed the patches on author-controlled GitHub Pages sites and a locally hosted clone of the Chinese developer site CSDN, then ran them against five GUI-agent backends across 600 online test cases. The patch reached the vision-language model's output 84.5% of the time. It survived the agent's action-parsing step 47.0% of the time. And it produced a verifiable, real-world consequence, a command that actually ran, 20.3% of the time.
That 20% number matters more than it looks. These are agents that take screenshots, decide what to click, and act on a live desktop or browser. If an attacker can plant one image that gets a fifth of attempts to execute arbitrary commands, anyone deploying these agents with real file access or shell access is exposed.
The trajectory data is what should sit with you. The paper's own analysis shows the agent running the injected command, then resuming the benign task the user actually asked for. A person glancing at the screen sees the agent doing its job. They don't see the extra step that happened first, and there's no changed cursor, no popup, no pause, just the same task finishing on schedule.
The paper doesn't say whether commercial agent products (the browser and desktop agents shipping from major labs right now) use the same five backends tested here, or whether they've since added defenses against patch-based prompt injection. That gap is the open question. Until vendors say otherwise, treat any agent with terminal or file-system access as one crafted image away from doing something you didn't ask for.
Each link below shares sources, entities, or timing with this story.
ADeptS-Bench tests seven models on paired benign and malicious GUI tasks and finds none clearing 80% task success while staying under 30% attack success. The ablation is the usable finding: removing the refusal tool raises attack success 21-23pp for tool-dependent models and 1...
Alibaba Tongyi Lab's technical report describes a foundation GUI agent spanning mobile, computer-use, web and DeepSearch, with a unified action space interleaving GUI operations with CLI execution and emitting batched actions per model turn. 82.1% MobileWorld, 92.2% MobileWorl...
Salesforce AI Research argues in arXiv 2607.22798 that a screenshot is a lossy rendering of program state — different states produce identical pixels — so the main agent should manipulate files, backends, and the DOM through code, delegating to a GUI subagent only when necessa...
CONFLICTGUI benchmarks conflict-aware termination, covering instructions that contradict themselves and instructions that contradict what's on screen, built on the observation that real users issue infeasible instructions by ordinary mistake. The result is execution-biased ove...
OSReward builds human-verified ground truth for computer-use trajectory judgments and finds even state-of-the-art models fall short with a consistent bias toward misclassifying failures as successes. The authors release OS-Shepherd at 9B and 35B, trained on a 100K corpus, clai...
arXiv 2608.04755 injected Android permission popups into real GUI tasks across four frontier multimodal LLMs with synchronized screenshots and UI trees. Holding the task fixed and changing only the requesting app flipped grants from 26/32 to 0/32, an App-Trust Bias. Holding th...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.