Continual post‑training of large multimodal models must add new capabilities while preserving those learned during pre‑training, a tension between two opposing goals. Supervised fine‑tuning (SFT) offers explicit target supervision that can learn a task from near‑zero accuracy, but its off‑policy targets push the model far enough to cause forgetting. On‑policy approaches such as RLVR and self‑distillation keep the policy close yet provide little signal when the policy cannot yet solve the task.
We introduce ReDraft (Reference‑Driven Revision and Fine‑Tuning), which extracts both benefits from the model’s own failures: an expert response is used only as a reference, the model revises its incorrect rollout, the revision is kept if a verifier accepts it, and fine‑tuning proceeds on the surviving targets. Each retained target is therefore explicit while remaining close to the current policy.
Across Counting, Clock Reading, and Jigsaw on Qwen2.5‑VL‑3B/7B—two tasks starting with near‑zero accuracy—ReDraft achieves 56.9 points on the target task versus 52.9 for SFT, while reducing prior‑task loss from 16.6 to 1.5 (an 11.3× reduction in forgetting). It also improves on OPSD along both axes (gain 19.3, loss 6.2).
Data‑ and parameter‑space analyses confirm the design: revised targets have higher probability under the base model, the induced updates are compact, and they follow SFT’s direction more closely than OPSD’s. Repairing the model’s own output rather than replacing it with an expert’s answer enables a single objective to achieve both learning and retention.
Review