Reinforcement Learning with Verifiable Rewards (RLVR) trains a language model by sampling responses, scoring them with an automatic verifier, and updating the policy based on the resulting rewards. The policy only receives learning signals from trajectories that appear in its own rollouts; under a fixed rollout budget, repeated sampling from the same policy tends to concentrate on similar reasoning paths, leaving useful alternatives unexplored.
To address this limitation, we introduce Activation‑Space Exploration (ASE), an online method that diversifies training rollouts. ASE first clusters activation differences between correct and incorrect answers to the same problem, building a structured bank of latent exploration vectors. These vectors are then added to the model’s activations in subsequent rollouts, steering the model toward alternative reasoning trajectories.
Experiments on Qwen3‑1.7B‑Base and Qwen3‑8B‑Base models trained on the DAPO‑Math‑17k dataset show that, with the same rollout budget, ASE improves the average Mean@8 across six mathematical reasoning benchmarks by 4.87 percentage points and 1.87 percentage points over GRPO, respectively.
Review: ASE demonstrates that systematic manipulation of activation space can effectively increase reasoning diversity under limited sampling budgets, offering a practical exploration strategy for RLVR deployments.