On‑policy self‑distillation (OPSD) enables a language model to learn under the guidance of a Teacher instantiated from the same model, where the Teacher has access to privileged information such as a reference solution. Sufficient privileged information allows the Teacher to produce highly accurate solution trajectories, but accuracy alone does not guarantee effective Student learning. Directly providing a complete solution often leads the Teacher to generate short, reference‑conditioned reasoning that omits exploration, verification, and self‑correction—processes essential for a Student that lacks privileged information.
To address this issue, we propose TeachBack, a bidirectional preference‑learning framework. TeachBack shapes the Teacher’s generation by controlling the length of the reference‑reasoning suffix injected into the prompt and by varying the prompt style. This encourages the Teacher to explore more thoroughly and re‑solve problems while preserving comparable accuracy, thereby reducing reliance on the reference solution.
At each rollout refresh, a shared policy simultaneously generates two trajectories: a privileged Teacher trajectory and a problem‑only Student trajectory. A pairwise preference objective then uses the privileged Teacher trajectories to improve Student learning, while a reversed preference leverages the Student’s behavior to steer future Teacher supervision under privileged conditions.
Experiments on mathematical and logical reasoning tasks demonstrate that TeachBack consistently yields higher Student performance compared to existing OPSD methods and related baselines.
Review