Knowledge distillation aims to let a compact student model emulate a large teacher model in resource‑constrained settings. Traditional distillation, however, is data‑hungry and computationally costly, especially for generative tasks. Few‑shot distillation transfers teacher knowledge using only a handful of examples, but its success hinges on the quality of those examples; samples near the teacher’s decision boundary that are highly uncertain tend to degrade student performance.
To address this, we propose COFFEQA (Counterfactual‑guided Few‑shot Distillation in Open‑book Question Answering), which injects counterfactuals to supply extra supervision around challenging factual samples. In the generative QA setting, a counterfactual is defined as a minimally altered context that forces the teacher to switch its answer to a designated alternative. Geometrically, factual‑counterfactual pairs constrain the deviation between the teacher’s and student’s decision boundaries.
We also introduce Context Following Difficulty (CFD), a metric that validates counterfactuals before distillation. CFD scores a pair based on (1) whether the teacher’s answer under the modified context matches the intended alternative answer, and (2) whether the answer change truly depends on the context modification. Only CFD‑validated pairs are used for few‑shot distillation.
Experiments on five in‑domain and three out‑of‑domain QA benchmarks, using the T5 and Qwen2.5 model families, show that COFFEQA consistently outperforms factual‑only baselines under the same few‑shot budget. For example, on TriviaQA with T5 and 32 few‑shot pairs, COFFEQA improves the baseline by 8.77 EM points and 9.98 F1 points.
Review: COFFEQA leverages CFD‑filtered counterfactuals to provide targeted supervision for few‑shot distillation, markedly boosting student performance on open‑book QA and highlighting the promise of counterfactual reasoning in model compression.