As long-horizon AI agents become increasingly capable, user interaction with these agents remains surprisingly thin. Typically, users provide an initial instruction and receive only selective textual updates, resulting in a lack of clarity regarding the agent's ongoing activities and when to intervene. This highlights a missing component in the current agent ecosystem: an always-on Jarvis-style mediator that maintains continuous real-time spoken interaction with the user, answers questions without disrupting the workflow, proactively reports progress or confusion, and injects user guidance back into the agent's execution when useful.
To address this, we introduce JarvisBench, a benchmark designed to measure the dual value of mediation in long-horizon agent workflows. JarvisBench features two complementary tracks: an agent-collaboration track that assesses whether mediation enhances downstream task completion, and a user-interaction track that evaluates whether mediation makes ongoing execution more understandable, responsive, and accessible to users.
We instantiate the benchmark with a modular reference Jarvis prototype and evaluate it on 34 text-only WildClaw tasks executed in OpenClaw. Preliminary results with GPT-5.5, Claude Opus 4.7, Gemini-based, and GPT-based worker agents indicate that Jarvis-style mediation can provide trace-grounded responses to user questions and enhance task performance when sparse user guidance is injected at appropriate moments. The findings also show that effectiveness heavily depends on the mediator's LLM brain, highlighting both the promise of this missing middle layer and the need for broader community effort. Demo page available at Jarvis Demo.
Blogger's Review: The introduction of JarvisBench offers a significant solution for user interaction in long-horizon AI agents, emphasizing the critical role of mediation in enhancing user experience and task completion. As technology evolves, such systems will better serve complex workflows, warranting further exploration and practice within the industry.