As coding agents take on more autonomous real‑world tasks, determining whether their risk‑treatment actions are justified becomes crucial. Existing studies assess related behaviors from isolated angles but lack a unified framework. To address this gap, we introduce ParanoiaEval, the first benchmark that evaluates risk‑treatment capabilities of coding agents in a unified manner.
ParanoiaEval is grounded in the well‑established Avoidance‑Transfer‑Mitigation‑Acceptance framework from software‑engineering risk management. We operationalize its four fundamental treatments for agentic coding scenarios and construct 200 evidence‑controlled repository‑level task pairs, each differing only in the evidence that defines the treatment.
Dedicated metrics are defined to capture risk‑treatment violations and evidence responsiveness, and a human‑calibrated agentic judge is employed for reliable evaluation.
Large‑scale experiments on eight representative models, complemented by a post‑hoc human study, reveal three key findings: (I) unnecessary risk treatment occurs in 11.2%‑58.7% of runs despite explicit evidence, with substantial variation across agent configurations; (II) higher task capability does not guarantee more appropriate risk treatment, and violations significantly degrade developer experience, establishing risk treatment as an independent capability dimension; (III) agents exhibit systematic patterns consistent with established risk‑management research, indicating that human practice knowledge can guide diagnosis and improvement of this capability.
Review: ParanoiaEval provides a reproducible benchmark for systematically assessing coding agents' risk‑treatment behavior, exposing current models' blind spots in defensive work and offering clear directions for future model enhancements.