Large language models are increasingly deployed as tool‑calling agents, yet safety evaluations still focus on their ability to refuse harmful requests in pure text. We construct a benchmark of 2,304 prompts across four domains, each presented as a triplet: text‑only, standard tool‑enabled, and adversarially framed tool‑enabled, covering both harmful and benign requests. Using 20 predefined forbidden actions, we test five models of varying sizes and families for tool‑call safety. The experiments reveal that many requests refused in chat are still executed unsafely when tools are enabled, with the divergence ranging from 10% to 52% under standard prompting. To probe the underlying mechanism, we identify a refusal‑associated direction in the residual stream of each model and observe that its activation decreases in four of the five models when tools are enabled. Lower activation predicts unsafe tool calls across all models (AUC 0.76‑0.86). Restoring the signal to the text‑only level via activation patching only prevents a subset of unsafe calls, mainly by suppressing tool use rather than producing safer calls. Steering strongly along the refusal direction nearly eliminates unsafe calls but also suppresses benign tool use or degrades output quality. These findings indicate that the refusal direction influences whether a model acts, yet it only partially explains the safety gap between text and tool contexts. Consequently, safety assessment of tool‑calling agents must directly examine their actions alongside textual responses.
Review