A verified reference solution supplies a correct trajectory for training a reasoning model. Alternatively, a prefix of that reference can guide the model to generate its own continuation. The central question is how much reference guidance to provide. We investigate this via prefix continuation: the model continues from a reference prefix and keeps the resulting trajectory if it passes verification, otherwise it falls back to the reference. Both procedures yield correct trajectories, so we compare their distributions to the ideal distribution—the model’s own distribution conditioned on successful verification. For a single continuation we derive a closed‑form KL divergence: $$KL = D_{KL}(p_{cont}\|p_{ref}) = -\log P_{ref}(suffix\mid prefix) + \text{bounded term}$$ The expression shows that, up to a bounded term, KL decreases with the product of the probability of generating a different correct trajectory and the reference surprisal (the negative log‑probability of the reference suffix given the prefix). A longer prefix tends to increase the former but decrease the latter, so continuation success alone does not dictate the optimal amount of guidance. From this analysis we construct a prefix selector shared across training questions, learned directly from continuation outcomes without estimating success probabilities or performing extra generation. The resulting Adaptive Reference Guidance (ARG) builds correct trajectories within a fixed generation budget and is applied to all‑failure groups in Group Relative Policy Optimization (GRPO). Experiments on Qwen3‑4B and Qwen3‑8B across five mathematical reasoning benchmarks demonstrate that ARG achieves the highest aggregate pass@12 among evaluated methods while maintaining competitive average sampled accuracy.
Review