NeFut Logo NeFut
中 Admin Login

[CS.AI] SteerDistill: Quantizing LLM Agents via Closed‑Loop Teacher Steering

Published at: 2026-11-11 22:00 Last updated: 2026-11-12 00:51
#Machine Learning #optimization #LLM

Low‑bit quantization can dramatically cut the inference cost of LLM agents, yet it often harms task success. Analyzing failed rollouts reveals that quantized agents tend to ignore task requirements or environmental constraints, producing state transitions that diverge from full‑precision references. Offline data usually miss the student’s failure contexts, and the student’s own rollouts provide few correct responses in those contexts, making behavioral correction difficult.\n\nTo address this, we introduce SteerDistill, a quantization‑aware distillation approach. A full‑precision teacher intervenes when the student generates erroneous responses, steering the student to revise the failed steps and iteratively refining its guidance based on feedback from the student’s execution. This closed‑loop teacher steering yields locally verified behavioral corrections. Afterwards, we train the student without guidance so that it matches the steered student’s predictions at corrected turns and the full‑precision teacher’s outputs at other turns, enabling autonomous inference.\n\nExperiments span five agentic benchmarks and three model families. Under the W4A16 configuration, SteerDistill improves average scores by 1.67–2.18 points over the strongest quantized baseline and stays within one point of the BF16 reference. On $\tau^2$‑Bench, it surpasses QeRL by 1.84% in success rate while using 67% fewer training GPU hours and producing shorter successful trajectories.\n\nReview: SteerDistill’s closed‑loop teacher steering effectively corrects the behavior of quantized models, balancing efficiency and task performance, and offers a practical pathway for deploying low‑bit LLM agents.

Original Source: https://arxiv.org/abs/2611.13410

[h] Back to Home