Post‑training quantization reduces the deployment cost of language‑model agents, yet its impact on recovery from temporary tool failures depends on how recovery is evaluated. We compared 8‑bit and 4‑bit variants of Llama‑3.1‑8B‑Instruct and Qwen2.5‑7B‑Instruct on twenty deterministic tool‑use tasks using five prompts.
The 8‑bit vs 4‑bit recovery gap changes direction across prompts and evaluation targets. For Llama, the difference under the same prompt ranges from 0 to +20.2 percentage points; for Qwen it spans -50.0 to +35.0 points. Full‑pipeline point estimates favor 8‑bit Llama under all five prompts, while Qwen’s comparison flips with the prompt.
The evaluation target can also reverse the result. For Llama under one prompt, scoring each variant only on its own clean‑passing tasks gives a 4‑bit advantage of 17.5 points; scoring the same tasks for both yields no difference; scoring the full pipeline favors 8‑bit by 28.3 points.
Executor leniency is a third factor. Rescoring the same logs with strict output parsing, which 8‑bit Llama violates far more often than 4‑bit Llama under that prompt, turns the +28.3 into -15.0, while Qwen remains essentially unchanged.
These findings show that a single prompt, a single screened task set, or a single scoring policy does not provide a stable conclusion about quantized‑agent robustness. Evaluations should compare variants on matched tasks, report full‑pipeline success for deployment decisions, state the scoring policy, and quantify uncertainty across tasks rather than injected fault sites.
Review