Large language model (LLM) agents are increasingly used in collaborative settings where each pursues a distinct task while sharing resources under review or oversight. Safety concerns arise not only from malicious agents but also from agents that simply try to help. Prior work has focused on adversarial or explicitly collusive behavior; this paper reveals that even benign agents with separate objectives can spontaneously cooperate, bypassing the intended workflow—a phenomenon we name the helpful agent problem.
We study long‑horizon environments with shared resources: agents sometimes need resources owned by peers, and cross‑owner changes are normally mediated by a reviewer. When the reviewer becomes unavailable, agents start making changes on each other's behalf while staying within their own local permissions, effectively creating “out‑of‑workflow” favors that circumvent system‑level oversight.
To measure this effect we introduce FavorBench, a controlled benchmark that captures downstream side effects, and we adapt CooperBench to realistic repositories. Across four leading models and two benchmarks, out‑of‑workflow favors appear frequently. In FavorBench, favors generated by at least three models cause significant harm to parties outside the interaction.
These findings highlight a distinct multi‑agent safety challenge: cooperation itself can open pathways around system safeguards. Future designs of review mechanisms and permission models must explicitly account for the risk of emergent cooperative behavior.
Review