NeFut Logo NeFut
中 Admin Login

[CS.AI] Agent Harness Awareness: Turning Verified Telemetry into Operational Control for Tool-Using Agents

Published at: 2026-11-05 22:00 Last updated: 2026-11-07 12:32
#AI #Machine Learning #LLM

Tool descriptions only tell an agent what an action can do, but they do not guarantee that the action is safe or appropriate at the moment of use. Execution environments often know this through monitors, traces, and recent failures, while the model lacks that visibility; this gap is called operational blindness. We therefore propose Agent Harness Awareness: the host supplies a small, fresh, verified state snapshot before a consequential action and retains authority for monitoring, enforcement, evaluation, and final dispatch, achieving a trust‑separated design.\n\nWe evaluate the idea on the ToolRoute benchmark, which keeps the task and candidate semantics fixed while varying the operational information available to the model. In a 216‑decision ablation across GPT‑5.6 Sol, Claude Sonnet 5, and Gemini 3.7 Flash, adding verified tool‑health state reduced mean observable policy regret from 4252 ms to 131 ms (a 96.9% drop) and raised single‑dispatch success from 69.4% to 95.8%.\n\nIn a separate 27‑decision end‑to‑end test with local TCP/HTTP execution, OpenTelemetry client spans, and Toxiproxy‑injected faults, the verified state cut mean regret from 8956 ms to 0 ms (zero excess operational cost) and completed the live action in 9/9 model‑by‑fault cases versus 5/9 for the neutral control.\n\nThese findings demonstrate that the tools‑and‑network instance of Agent Harness Awareness can materially improve pre‑dispatch operational choice through host telemetry. Runtime state, hardware pressure, economics, retry‑aware recovery, and heterogeneous selection remain open empirical extensions of the architecture.\n\nReview: The study shows that injecting trustworthy runtime snapshots into large‑model tool use dramatically reduces decision risk and execution overhead, offering a practical path toward safer, more reliable autonomous agents.

Original Source: https://arxiv.org/abs/2611.05391

[h] Back to Home