Traditional evaluation of AI agents focuses on task completion, yet in interactive services agents that repeatedly ask questions, perform redundant searches, or make unnecessary revisions can frustrate users. To address this, we introduce RideWay, an efficiency‑centered benchmark for ride‑hailing agents operating in a stateful tool‑calling environment. The benchmark is paired with the Efficiency Utility metric, which, after confirming task success, penalizes excess tool calls and user‑facing dialogue turns relative to a task‑specific reference effort. Penalty weights are calibrated using extensive human‑paired preference data, capturing a service‑workflow trade‑off: extra dialogue often creates visible friction, whereas extra tool usage can sometimes verify constraints or preserve user intent. Experiments across 58 tasks and 24 models reveal that the fitted penalty for surplus turns is roughly twice that for surplus tool calls. On held‑out, task‑disjoint preference tests, Efficiency Utility attains 78.7% overall accuracy—90.6% when trajectories differ in turns, but chance‑level performance when they differ only in tool calls—highlighting the weakest human agreement on tool‑use differences. RideWay thus makes interaction efficiency measurable alongside task success and exposes the limits of count‑based tool‑use evaluation.
Review: RideWay offers a fresh perspective on assessing language agents in real‑world services, emphasizing the balance between efficiency and success, and provides valuable guidance for scenarios that heavily rely on external tool invocation.