This paper introduces tool efficiency, a new quantitative metric to evaluate the rate of useful tool calls in an LLM agent trajectory. To ensure that tool efficiency is well-defined, we also introduce marginal tool utility, a new quantitative metric defined per tool call indicating whether a tool is useful or whether it can be safely removed from the tool suite without affecting accuracy while increasing tool efficiency. We determine the sign of marginal tool utility for each tool call in a trajectory using LLM-as-a-Judge.
While much prior work has been done to develop techniques that improve tool use by LLMs and design evaluation methods measuring efficiency indirectly using accuracy as a proxy, our work is centered on measuring efficiency directly via the quantitative metric proposed in this paper in post hoc trajectory analyses.
It is our intention that this work contributes to the frontier of LLM evaluation research as a springboard for future benchmark designs and agent harness engineering (specifically regarding creating lean tool suites) that optimize for metrics that complement but are distinct from accuracy.
Blogger's Review: The concepts of tool efficiency and marginal tool utility presented in this paper provide a new perspective on performance evaluation for LLMs, highlighting the direct impact of tool selection on efficiency. Future research should focus on quantifying the actual utility of tools, enabling better design of efficient tool suites and enhancing overall LLM performance.