Group Relative Policy Optimization (GRPO) is a widely studied reinforcement‑learning method for tasks with verifiable rewards. Its advantage estimator assigns each rollout a magnitude based on within‑group reward statistics, typically rewarding rollouts that reach the correct answer through reasoning.
An overlooked situation occurs when a rollout arrives at the answer by guessing; the same formula still yields a high magnitude, which we call a spurious advantage. This phenomenon appears in three settings: bounded‑answer tasks with a small candidate set, open‑answer tasks that contain bounded sub‑cases, and search agents whose budget opens many paths to the same answer.
In all three cases the policy is misled toward guess‑like behavior. To address this, we propose SIGNBALANCE. Its design principles are: keep the verifier sign unchanged, use a global scale instead of per‑group statistics, and restore zero‑mean balance via a stop‑gradient per‑class rescaling. The resulting magnitude can be written as $$\hat{A}=\text{sgn}(v)\cdot\frac{g}{\|g\|}\,,$$ where $v$ is the verifier output and $g$ is a global gradient vector.
Across a suite of math and search‑agent benchmarks at various scales, SIGNBALANCE matches GRPO on open‑answer math and yields clear improvements on bounded‑answer math and search agents. The code will be released.
Review