Large language models (LLMs) are unreliable at arithmetic, which is critical for clinical calculators where a single numerical error can change a medical recommendation. The usual remedy is to hard‑code each calculator as a validated function. This work investigates an alternative: the model does not perform the calculation itself; instead it generates case‑specific Python code that a restricted local executor runs deterministically. Consequently, the model’s role reduces to deciding how to invoke the generated code.
We evaluate this Program‑Solve interface on the MedCalc‑Bench Verified dataset (1,100 cases, 55 calculators), comparing it against direct model arithmetic and a hand‑written library covering 22 calculators. The models used are Qwen2.5‑7B and Qwen2.5‑32B‑AWQ. First, we audited the benchmark formulas against current clinical guidelines, flagging 16 of the 55 calculators for version, usage, or coefficient concerns.
When formulas and gold‑standard variables are supplied and both approaches read the full clinical note, handing off to the solver yields no reliable advantage for the 7B model: accuracy 75.31% vs. 72.02% for direct arithmetic, a paired gain of +3.29 points with a 95% calculator‑cluster interval of [-3.49, 10.38]. For the 32B model the advantage is clear: accuracy 90.53% vs. 83.47%, +7.05 points, interval [0.47, 14.60], clearly above zero.
The hand‑written library is exact on its 440 supported cases but abstains on the rest, resulting in an overall coverage of only 40.0%. Thus, adding an executor helps some open‑weight models more than others even under matched formula, variable, and note access, but it does not replace the need for verified formulas or reliable variable extraction.
Review: The program‑generation plus deterministic solving pipeline shows promising gains for larger LLMs, yet its practical deployment still hinges on rigorous formula validation and robust variable extraction.