In reinforcement learning with verifiable rewards (RLVR), despite its widespread use in enhancing the reasoning capabilities of large language models (LLMs), the generalizability of the resulting models remains poorly understood. This work establishes the first non-vacuous generalization bounds for parameter-efficient RLVR fine-tuning at the billion-parameter scale. Our approach adapts PAC-Bayes compression bounds to this setting and addresses the inherent stochasticity of token generation by applying the Gumbel-max reparameterization trick.
To operationalize these bounds, we propose the Progressive RLVR framework, which integrates RLVR with on-policy distillation, TinyLoRA, and model quantization. The Progressive RLVR empirically retains 84-97% of the performance of standard LoRA fine-tuning while producing models that are 14,796x more compressible. We show that this framework yields non-vacuous generalization bounds in four domains: mathematical problem-solving, programming, general-knowledge reasoning, and Text-to-SQL. Our bounds exceed the base model's accuracy by 9-51% and lie within 6-11% of the accuracy of the fine-tuned models.
Blogger's Review: This paper introduces the Progressive RLVR framework, making significant strides in establishing generalization bounds in reinforcement learning with verifiable rewards. The remarkable advancements in model compression while maintaining performance are noteworthy, providing a solid foundation for understanding the generalization capabilities of RLVR in large-scale language models.