As agentic systems are rapidly deployed in safety‑critical domains, measuring the confidence of their actions becomes crucial. Compared with conventional machine‑learning pipelines, agentic workflows involve planning, tool invocation, and dynamic environment interaction, leading to more intricate failure modes. This work investigates whether a model's internal representations can offer stronger signals of eventual task success than surface‑level generation.
We introduce two complementary techniques:
- Latent Trajectory Dynamics (LTD) aggregates changes in residual‑stream representations along an interaction trajectory, capturing hidden success trends.
- Action Representation Probe (ARP) predicts success directly from the representations formed at action decision points.
Across three interactive benchmarks (Bash, SQL, Python) and three model families (Qwen14B, Qwen7B, DeepSeek6.7B), both methods consistently outperform surface‑level generation and sequence‑based calibration baselines. The monitor incurs zero overhead, requiring neither prompt modifications nor multi‑sample rollouts.
Review: The paper delivers a lightweight, model‑agnostic confidence calibration approach that can be readily integrated into safety‑critical agent deployments.