This work introduces an end‑to‑end evaluation framework for image‑triggered command injection against computer‑use agents (CUAs). The framework tests whether a local visual patch can cause verifiable environmental effects across the full chain of screenshot input, vision‑language model (VLM) generation, action parsing, and environment execution. We train and deploy patches on author‑controlled GitHub Pages and a locally hosted CSDN clone, then evaluate them in real settings on five open‑source or publicly available GUI‑agent or VLM backends. Across 600 online instances the metrics T‑ASR, TAPR, and E2E‑ASR reach 84.5 %, 47.0 %, and 20.3 % respectively. Trajectory analysis reveals that in some successful cases the agent first runs a malicious terminal command and then resumes the original benign task. These findings indicate that optimized local visual signals can affect VLM outputs and propagate through the execution pipeline of open CUAs, posing real environmental risks.
Review