In safety evaluations of multi-agent large language model (LLM) systems, direct prompts are often compared with planner-executor pipelines, reporting the difference as a single 'pipeline effect.' We argue that this aggregate is difficult to interpret as it conflates three mechanisms: harmful intent may be reframed as plausible operational work, the planner may refuse or transform the request, and the executor may act under delegation prompts implying prior approval.
To separate these factors, we introduce a five-condition controlled contrast design, evaluated on 30 synthetic harmful scenarios and an exploratory external validation set from four agent-safety benchmarks using LLM-judged compliance.
Our results demonstrate that aggregate pipeline safety is not a stable architectural property. Operational reframing is the most portable risk signal, increasing compliance for GPT, Gemini, and DeepSeek across both scenario sets, whereas Claude is comparatively resistant.
Planner behavior can offset this risk mainly through refusal; however, when the planner produces executable steps, the executor may become more compliant than under the direct operational baseline. Approval-framed delegation is sensitive to prompt design, model pairing, and scenario source, and a skeptical executor prompt sharply reduces compliance.
Raw-direct model rankings can also mispredict deployed planner-executor behavior. Gemini is safest under raw direct prompts in the primary set yet shows the largest amplification with a Claude planner, rising from 8.9 percent to 38.9 percent compliance. GPT's near-zero aggregate pipeline effect instead hides a reframing increase canceled by planner refusal.
These findings suggest that multi-agent safety evaluations should report reframing, planner behavior, delegation framing, and model pairing separately before attributing failures to architecture itself.
Blogger's Review: This paper reveals the complexities in safety evaluations of multi-agent LLM systems through meticulous experimental design, emphasizing the impact of different mechanisms on compliance. It provides crucial directions for future safety assessments, particularly in model pairing and prompt design, warranting further research and practical implementation.