Abstract
Sampling multiple solutions and returning the majority answer is one of the most reliable methods to enhance the reasoning accuracy of large language models without labels. A growing number of methods convert this consensus signal into training supervision. However, existing approaches utilize consensus only in restricted forms: as a filter for selecting solutions for fine-tuning, as a preference between answers, or as a scalar reward for reinforcement learning, discarding much of the information contained in agreeing solutions.
We present CANON (Consensus-ANchored self-distillatiON), a label-free training method that transforms consensus into dense, token-level supervision. For each unlabeled prompt, CANON samples multiple solutions, extracts the majority answer, and conditions a frozen snapshot of the model on the solution that achieves it; this consensus-anchored teacher then supervises the model on its own rollouts at every token.
Experiments on mathematical and scientific reasoning benchmarks demonstrate that CANON improves pass@1 by up to 12 points, outperforming label-free reinforcement learning by 6 points with only a seventh of its compute, and approaching a teacher conditioned on gold solutions. Trained on pooled unlabeled data, it transfers to held-out benchmarks, matching training methods that utilize gold labels. Analysis suggests that the improvements are not merely distribution sharpening: after training, the model solves problems it previously could not in 32 attempts, and its majority vote itself becomes more accurate.
Blogger's Review: The CANON method effectively leverages consensus information to enhance label-free learning performance, showcasing high research value and application potential. Particularly in resource-constrained scenarios, this innovative approach can significantly improve the reasoning capabilities of models, warranting further exploration and practice.