Adding an abstention option to a task‑adapted multimodal model often discards correct answers, including capabilities acquired during adaptation. We formulate abstention repair as maximizing correct abstention on unanswerable questions while staying within a budget on newly lost correct answers. To this end we propose ReFrain (Regression‑Free Abstention), which consists of three components: a linear probe on the first‑token logits to assess answer confidence, threshold calibration on held‑out image clusters that guarantees zero loss on the calibration set at the zero‑budget target (hence “regression‑free”), and a deterministic veto that replaces rejected outputs with the displayed “none of the above” option while leaving all other responses untouched.
All baselines start from the same answering‑SFT checkpoint; we separately track lost correct answers, repaired errors, and retained SFT gains. Experiments on Qwen2.5‑VL‑3B show that at zero budget ReFrain preserves all 57 SFT‑gained answers and improves correct abstention by 7.30 percentage points. With a 0.5 % calibration budget it reaches 74.05 % correct abstention, incurs only 8 losses and retains 55 gains; by contrast CLIP‑UP EmbLoRA attains 64.65 % with 62 losses, 41 repairs and 40 retained gains. The same probe at its default threshold loses 298 correct answers. When models and thresholds are frozen, validation on 228 independent question clusters yields 66.05 % correct abstention with no losses at the 0.5 % point, surpassing EmbLoRA by 32.72 percentage points. Ordinary refusal SFT can achieve higher abstention but at a greater regression cost. Code and reproduction materials are available at https://github.com/ModeLiberaL/ReFrain.
Review: ReFrain demonstrates that a lightweight linear probe combined with careful threshold calibration can achieve a zero‑regression objective, substantially boosting safe abstention in multimodal settings while virtually preserving existing correct answers, highlighting a promising direction for balancing safety and performance.