Reliable monitoring of web agents becomes difficult when internal uncertainty signals such as token logits are unavailable. This work tackles prefix‑level risk prediction: given an evolving execution prefix, estimate whether the current run remains on track or is drifting toward failure. We introduce two observable trajectory representations:
- Macro features: summarize cross‑step agent‑environment behavior and feedback, capturing the overall trend.
- Micro features: repeatedly query the black‑box model to measure consistency among intention, action, and anticipated state change.
Instead of inheriting the final outcome label, we define a key‑step boundary: the first critical error in the observed continuation that stays uncorrected and is linked to the final failure. This preserves valid early prefixes of failed trajectories as “on track”, avoiding premature failure labeling.
Experiments on the WebArena‑Lite and Online Mind2Web benchmarks, using five open‑ and closed‑source backbones, show that observable trajectory signals are competitive with internal‑signal baselines. Moreover, the predictors enable:
- Early intervention under a fixed false‑positive budget.
- Transferability to unseen website categories.
These results demonstrate that risk prediction can be effectively supported by external observable signals alone, offering a practical avenue for safe deployment of web agents.
Review