Abstract
Weak-to-strong generalization offers a promising paradigm for scaling large language models (LLMs) by training stronger models on samples from aligned weaker ones, without requiring human feedback or explicit reward modeling. However, its robustness and generalization are hindered by the noise and biases in weak-model outputs, limiting its applicability in practice.
To address this challenge, we leverage implicit rewards, approximating explicit rewards through log-likelihood ratios, and reveal their structural equivalence with Contrastive Decoding (CD), a strategy shown to reduce noise in LLM generation.
Building on this connection, we propose the Contrastive Weak-to-Strong Generalization (ConG) framework, which employs contrastive decoding between pre- and post-alignment weak models to produce higher-quality samples. This approach enables more reliable capability transfer, denoising, and improved robustness, significantly mitigating the limitations of traditional weak-to-strong methods.
Empirical results across different model families confirm consistent improvements, demonstrating the generality and effectiveness of ConG. Overall, our findings highlight the potential of ConG to advance weak-to-strong generalization and provide a promising pathway toward AGI.
Blogger's Review: The Contrastive Weak-to-Strong Generalization (ConG) framework effectively reduces noise in weak model outputs through the introduction of contrastive decoding, thereby enhancing overall model performance. This method offers a fresh perspective on training large language models, particularly in scenarios without human feedback, showcasing its significance in the future development of AGI. Notably, the universality and adaptability of this approach will provide new insights for subsequent research.