During supervised fine‑tuning (SFT), a model’s performance on the target domain often improves at the expense of its ability to access out‑of‑distribution (OOD) facts. The authors label this phenomenon “factual access failure”: the model can still recognize or rank the correct answer under constrained evaluation, yet fails to produce it in closed‑book generation.
Benchmark comparisons, same‑fact multiple‑choice and generation probes, and failure‑mode analysis reveal that the degradation comprises genuine wrong‑answer generations as well as expression‑level issues such as verbosity, formatting mismatches, and over‑reliance on exact‑match artifacts.
To address this, they propose Recall‑Anchored Distillation (RAD), a base‑anchored self‑distillation objective that aligns the adapted model’s soft continuation distribution with that of the original base model on unlabeled OOD text. RAD requires no gold OOD answers, external judges, or labeled factual data.
Experiments on three backbones fine‑tuned on MedMCQA show that RAD recovers a consistent portion of the lost OOD recall while preserving target‑domain adaptation. Compared with simple replay of the same OOD text, the key preservation signal is the base model’s soft distribution rather than additional text exposure alone.
Blogger's Review: RAD cleverly leverages the implicit knowledge of the base model, mitigating factual access failures without extra annotation costs, and offers a practical solution for real‑world LLM deployment.