Compact instruction‑following rerankers are attractive for deployment, yet conventional distillation pipelines train students by offline imitation of teacher outputs on a fixed example set, limiting supervision to the teacher's observed ranking space. We revisit reranker distillation from a reinforcement‑learning perspective and propose a two‑stage framework that combines off‑policy teacher optimization with on‑policy student distillation.
Stage 1 strengthens a 4B teacher reranker using off‑policy GRPO with LLM‑judge feedback on 88K instruction‑following examples.
Stage 2 lets a compact 1B student sample rankings from its own policy and receive soft teacher‑derived rewards on those rankings, coupling student exploration with knowledge transfer.
The strongest gains appear under distribution shift. On MAIR‑11 (the original 11‑subset, 869 queries), the student reaches 0.7670 nDCG@6, outperforming offline listwise KD by +4.6 points. Controlled comparisons against offline pairwise RankNet KD and on‑policy GKD show that neither changing the offline distillation objective nor matching teacher distribution on‑policy reproduces the performance of reward‑based on‑policy distillation over student‑sampled rankings. The advantage persists on MAIR‑Full: across 126 tasks and 9,356 queries, the method obtains the highest task‑macro estimates, 0.6808 nDCG@6 and 0.7865 MRR@6. It also exceeds two released 7B RL‑trained rerankers on the comparable MAIR‑11 evaluation, while the same Stage 2 procedure consistently improves three architecturally distinct student backbones.
On a 9,861‑query validation benchmark, the resulting 1B reranker achieves 0.7624 nDCG@6 and offers a favorable quality‑efficiency trade‑off relative to larger alternatives.
Review