As coding agents become increasingly integrated into system operations, their tool usage can directly modify project artifacts, execution environments, and the underlying systems. For instance, if a coding agent inserts a hook into a system startup or configuration script, that change can persist after the interaction, be triggered later, and abuse delegated user or system privileges to modify the system. This positions security testing as a systemic problem: the critical question is not just what the agent says, but what it actually does to the surrounding environment.
We present an execution-grounded red-team testing framework for probing this execution-layer security boundary using observable sandbox evidence, including tool invocations, runtime traces, and file-system diffs. Our framework embeds target unsafe operations into routine software engineering workloads, including unit testing, regression testing, crash reproduction, and validation, and employs an execution oracle to guide refinement when an initial probe is rejected or fails.
Across multiple agent frameworks and model backbones, our red-team workload reformulation substantially increases verified unsafe execution, achieving 73.61% on code carriers and 53.93% on text carriers. These results indicate that coding agents in system operations remain insecure under task disguise: once risky intent is concealed within plausible engineering tasks, the agent can be induced to perform unsafe actions on the surrounding system. More broadly, coding agents in system operations still require stronger security testing and safeguards.
Blogger's Review: This research highlights potential security risks posed by coding agents in practical operations, emphasizing the importance of execution-grounded security testing in software engineering. By embedding unsafe operations into routine workloads, the authors effectively increased the detection rate of vulnerabilities, providing valuable insights for future security measures. Their methodology merits further exploration in other domains.