Reasoning models can improve performance by extending inference steps at test time, yet when additional reasoning yields no benefit or even harms accuracy, the phenomenon known as overthinking arises. Existing evaluations only compare full‑thinking accuracy and token consumption, lacking a metric for overthinking severity and a theoretical baseline for mitigation strategies.
To fill this gap, the paper introduces Step‑wise Truncation Probing (STP). Along each sampled reasoning trajectory, multiple prefixes are retained and each is used to generate an answer. The earliest position that attains the maximum empirical accuracy on that trajectory serves as a reference. Two metrics are defined:
- Overthinking Ratio $\text{OR}=\frac{\text{tokens generated after the reference}}{\text{total thinking tokens}}$
- Accuracy Degradation $\text{AD}=\text{accuracy at reference}-\text{accuracy at full thinking}$
These quantify the proportion of wasted reasoning and the resulting drop in performance.
The study then unifies dynamic routing, fixed budgets, and dynamic early exit into a single theoretical framework, enabling the calculation of each termination strategy’s optimal mitigation potential under a normalized token budget.
Using the complete STP measurements, a token‑accuracy Pareto frontier is constructed for each strategy under the common budget, clearly illustrating the accuracy ceiling achievable for a given token cost.
To support reproducibility, the authors release OTBench, a reusable benchmark that automates the STP process. OTBench has generated 70.5 B reasoning tokens and collected probing data across six models and four benchmark groups.
Review: The work delivers a rigorous quantification of overthinking and a unified evaluation framework for termination strategies, offering both theoretical insight and practical tools for improving inference efficiency in large language models.