Reinforcement learning (RL) has become central to enhancing the performance of large language models (LLMs) on complex reasoning tasks. However, RL post-training is often studied in isolation from the pretraining that precedes it, leading to two fundamental questions: (1) How do pretraining choices (model size, data) influence the returns to RL compute? (2) What does RL actually do to the model? These questions are challenging to investigate in the standard LLM setting due to the vast and uncontrolled pretraining corpora, making it difficult to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive.
To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, performing supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that post-RL performance at a given RL compute level is well-predicted from pretraining loss, and the slope of the RL reward curves improves approximately linearly with the number of pretraining tokens.
Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles, it amplifies correct moves already preferred by the SFT policy, while on hard puzzles, it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints achieve higher post-RL performance and improve faster under RL. In summary, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.
Blogger's Review: This article utilizes chess as a testing platform to deeply explore the relationship between pretraining and post-training, offering new insights into the reasoning capabilities of large language models. The findings indicate that RL not only enhances model performance but also reveals potentials that remain hidden in traditional training, highlighting its significant practical implications and research value.