Generative models increasingly expose a reasoning‑effort setting that lets users trade more inference‑time compute for potentially better performance. Users must pick an effort level before knowing whether the extra reasoning will help, and even a successful run does not reveal whether the purchased compute was necessary. Hence reasoning effort is a credence good: buyers see the outcome but cannot assess the marginal value of the computation they paid for.
We measure the counterfactual impact of effort levels on both proprietary and open‑weight models across four benchmarks, covering single‑call reasoning and long‑horizon agentic tasks. The experiments show that higher effort does not consistently preserve tasks solved at lower effort, can lead to overspending, and may alter the agents’ solution trajectories.
Specifically, while aggregate accuracy may rise with higher effort, many instances that succeeded with low effort fail under higher effort. For the subset of easy tasks solved by all effort levels, the highest effort can cost up to 42× the lowest. Moreover, higher effort makes agents verify their own work more heavily and search for solutions more often instead of proceeding directly, lengthening trajectories without necessarily changing the final answer.
These findings indicate that naively choosing the highest effort is both cost‑inefficient and offers no guarantee of best instance‑level performance. A simple alternative is to start at the lowest effort and only increase after a verified failure. This strategy matches or improves on fixed top‑effort performance across all models and benchmarks while reducing cost for most proprietary models.
Review