We argue that benchmarking large language models (LLMs) solely by accuracy overlooks a crucial deployment question: when does the cost of extra thinking tokens pay off? To address this, we introduce the Token Economy Score (TES), a marginal metric that captures the accuracy gain of a reasoning model over a non‑reasoning baseline, normalized by the generated‑token multiplier.
$$\text{TES}=\frac{\text{Acc}_{\text{reason}}-\text{Acc}_{\text{base}}}{\frac{\text{Tokens}_{\text{reason}}}{\text{Tokens}_{\text{base}}}}$$
For model families with a reasoning toggle, we define a paired TES that directly compares the same model with reasoning on versus off. For frontier models lacking a direct non‑reasoning counterpart, we propose an approximated TES, estimating the baseline from a closely related model.
We conducted 151 model‑benchmark runs across seven benchmarks covering mathematics, code generation, scientific reasoning, instruction following, expert knowledge, knowledge recall, and research‑level physics. The analysis focuses on three deployment‑oriented dimensions:
- Which task structures yield positive marginal reasoning efficiency;
- How increasing reasoning effort (e.g., more thinking steps) changes TES within model families;
- How deployment context (cloud vs on‑premises) alters economic viability.
Findings show that task structure predicts reasoning efficiency better than nominal difficulty. Sequential inference‑chain tasks such as AIME 2025 and LiveCodeBench achieve high TES, whereas seemingly hard knowledge‑recall tasks like MMLU‑Pro exhibit low TES. Moreover, higher reasoning effort leads to systematic diminishing returns, with cases where extra thinking actually reduces accuracy.
We also introduce Reasoning Cost Share (RCS) and Deployment Cost Multiplier (DCM). RCS reveals that internal thinking often dominates inference spend, while DCM demonstrates how on‑premises deployment can shift the economics of otherwise costly reasoning workloads.
Blogger's Review: This work offers a pragmatic economic lens for evaluating LLM reasoning, urging practitioners to enable reasoning selectively based on task type, effort level, and deployment setting rather than assuming it is universally beneficial.