Large language model (LLM) agents are increasingly deployed in long‑horizon interactions that involve tool use, persistent state, evolving authorizations, and feedback from external environments. In such settings, safety failures may only surface after several turns, yet most existing evaluations reduce agent behavior to task or attack success, obscuring whether the agent acts, refuses correctly, or stays properly calibrated as the interaction progresses.
We introduce Blindspot, a benchmark for trajectory‑level safety calibration of long‑horizon tool‑using agents. Blindspot evaluates complete user‑agent‑environment trajectories through adaptive adversarial interaction, stateful tool execution, and execution‑grounded adjudication. Its current instantiation comprises 22 attack families and 35 scenarios across seven domains, yielding over 2,500 long‑horizon trajectories with an average length of 14.7 turns. Each trajectory is assigned one of five outcomes: Safe Completion, Correct Refusal, Unsafe Completion, Over‑Refusal, or Indeterminate.
Unlike fixed attack datasets, Blindspot is an extensible live‑simulation framework where attacks, scenarios, tools, policies, domains, and agent configurations can be added without redesigning the evaluation pipeline. We evaluate 13 proprietary and open‑weight LLMs using eight metrics covering unsafe completion, appropriate refusal, benign utility, over‑refusal, repeated‑run robustness, and post‑refusal failure. Preliminary results reveal substantial differences in safety‑utility calibration across models and show that failures can emerge only after several initially safe interaction steps.
These findings motivate treating agent safety as a trajectory‑level property rather than a single‑turn or binary success criterion.
Review