Traditional agentic red-teaming benchmarks typically report the attack success rate as a binary value: either the attack succeeded or it did not. However, this binary metric discards crucial information that defenders require, specifically the degree of harm caused by the resulting action.
To address this, we introduce an action-graded harm rubric that scores an agent's tool-call trajectory on a seven-level ordinal scale (L0 to L6) based on criteria such as whether the executed action was reversible, whether it crossed scope to affect another party, and whether it expanded privilege.
We compute this scale in two ways: a deterministic oracle that reads the trajectory and the attacker's stated goal, and a panel of three frontier language model judges who read a tag-free account of the same trajectory.
Across four victim models and two defenses in the AgentDojo workspace, severity grading uncovers three cases that the binary metric obscures, including a defense that reports a zero attack success rate while still allowing an externally visible cross-scope leak through an unfiltered tool.
The judge panel reproduces the oracle with high ordinal agreement (Krippendorff's alpha = 0.91) but shares systematic blind spots, notably a failure to recognize escalation chains.
Unlike previous works that provide harm taxonomies, harmful-task completion tests, execution-level safety benchmarks, or severity-aware simulations, our contribution is a reusable, trace-grounded severity instrument applied to the actual actions recorded in existing red-team logs. All code, prompts, and per-episode logs are released.
Blogger's Review: This paper significantly enriches the assessment of AI agent attacks by introducing an action-graded metric that emphasizes the complexity and potential consequences of attack behaviors, providing a more nuanced reference framework for future security defenses.