In the recent arXiv preprint (2609.03966v1), the authors point out that conventional tool‑call rate metrics usually read only the output of the serving stack, ignoring the well‑formed calls actually emitted by the model. Because the interface censors the trajectory before downstream components see it, the reported rate can be zero even when the model generates complete calls.
Experiments on BFCL v4 data keep weights, cases, decoding settings and random seeds fixed while swapping only the serving adapter. The same model scores 0.00 versus 0.96/0.19 under different adapters. A 2×2 study with chat templates and parsers shows that all effects reside in the interface interaction itself—no individual component is defective, and fixing one side of the contract yields no overall gain.
On tau‑bench’s 115 interactive retail tasks, the adapter swap raises server‑parsed calls from 0 to 636 and tasks that actually trigger tool execution from 0 to 103. The phenomenon is reproduced across the Qwen2.5‑Coder family (0.2B‑32B): the server parses 0/100 calls at every scale, while well‑formed emitted calls reach 80/100 at 32B (≈72/100 after calibration against a gold standard).
For Llama‑3.1‑8B, the original model calls the task function itself 23% of the time, but this drops to 0 when the strict:true flag is enabled. The mismatch even leaks into the training loop: in verl’s AgentLoop, the 7B model produces complete calls in 45 of 115 generations, yet none are accepted, executed, or return an observation; the 1.5B model shows the same zero‑acceptance behavior.
During evaluation, repairing the adapter restores the parsing mechanism but does not significantly improve outcomes (parsing 0‑84, rescued 0‑9, final pass rate 53‑62, not statistically significant). The authors release a 98‑line pre‑flight check that catches every silent failure.
Conclusion: the observed tool‑call rate is not a property of the model alone; it is a property of the model‑interface stack that measures it.
Review