This study investigates whether RL post-training merely amplifies latent primitive skills in a base model or can compose these primitive skills into new higher-level strategies.
We conduct experiments in a fully observable rewrite-grammar environment, where the pretraining distribution is known and each generated rewrite can be audited. By post-training on a trace-based reasoning task, RL successfully solves problems that the pretrained model struggles with, even under larger sampling budgets.
The analysis reveals that RL reorganizes primitive competence through a phased compositional mechanism: it first strengthens primitive reductions, then discovers valid composed procedures. These procedures include sequential compositions, collapsing ordered chains of primitive contractions, and parallel compositions, combining independent primitive contractions in a single step. Notably, these composed procedures are not isolated samples; they are reused and consolidated into a stable repertoire.
Comparing RL with rejection fine-tuning (RFT), the key difference lies in selectivity rather than exploration volume. RFT generates numerous shortcut-like rewrites, many of which are invalid, while RL focuses exploration on valid reusable structures. Ablation studies indicate that the emergence of compositional strategies relies not solely on exposure to primitives, but on whether pretraining organizes primitive competence into reduction procedures that RL can later compress. The base model offers weak procedural ingredients, and RL builds them into reliable higher-level strategies.
Blogger's Review: This research highlights the potential of RL post-training in developing compositional reasoning strategies, emphasizing the importance of skill composition. This could have profound implications for the design of future AI systems, enabling models to tackle complex tasks more flexibly and warranting further exploration of their practical applications.