Abstract
Reinforcement learning with verifiable rewards, particularly Group Relative Policy Optimization (GRPO), is routinely run on a supervised checkpoint to develop stronger agents. This study investigates whether GRPO adds skills to a small language and vision-language model web agent at the 4B to 8B scale, or if it merely reshapes existing behaviors.
Across 18 runs varying learning rates, KL weights, seeds, initialization, and clipping, no configuration credibly improved the success rate of a strong supervised baseline on mastered tasks. In the text track, moderate to high learning rates significantly degraded performance. This null finding held under paired testing, 25 evaluation seeds, 6 training seeds, recipe changes, and both text and Set-of-Marks observations, with credible harm only noted in the text track when scaling to 8B.
To demonstrate that the null reflects the setting rather than a broken pipeline, we executed the identical harness, reward, and recipe on tasks with reachable rewards through sampling, where the success rate increased by 22 points with a paired interval excluding zero. Thus, GRPO only proves beneficial when there is room for improvement, indicating that the sampled policy already outperforms the greedy one.
We then elucidate the failure: a middle learning rate degrades the agent's performance, while a high rate collapses it, forming a double dissociation: degradation localizes to attention and MLP blocks, while collapse cannot be traced to any single group, with the embedding change dominating the weight movement being causally inert. In the 4B model, effective rank in late layers correlates with capability in both directions; however, in the 8B model, these aspects diverge. This coupling is specific to smaller models, thus reported as scale-dependent.
Blogger's Review: This paper delves into the failure mechanisms of GRPO in smaller models, revealing the complex effects of learning rates on agent performance. Through systematic experimental design, the authors effectively eliminate interference from other variables, demonstrating the limitations of reinforcement learning under specific conditions. This provides valuable lessons for researchers when selecting learning strategies.