Reinforcement learning from verifiable rewards (e.g., GRPO) serves as the backbone of today's reasoning models, but it only grades the final answer. This approach trains models to produce more rather than think better, as the reasoning trace itself is never graded, and there is no label for good thinking.
We introduce Agon, which enables two competing models to act as each other's graders. Both models tackle the same problem; in alternating roles, one drafts a solution while the other reads it during the solving process, and each is rewarded for out-solving the other. To win, a model must out-reason a rival that has seen its work, thereby implicitly judging reasoning during training without process labels or a reward model.
As both models are optimized, each faces an increasingly stronger rival, which is not achievable with single-model RL. They need only be comparably strong and behaviorally different. During inference, the pair operates as trained, in a two-stage cascade where one drafts and the other answers after reading the draft.
On the challenging DeepMath split with Qwen3, Agon doubles GRPO's pass@1, roughly eight times the gain of an untrained Mixture-of-Agents over the same base. This ordering replicates across competitive programming code and model families (Qwen3.5, Gemma 4). Currently, the models communicate in text; the next step is to enable them to reason together in latent space.
Blogger's Review: Agon breaks the limitations of traditional reinforcement learning by introducing a competitive mechanism that enhances the reasoning capabilities of models. This approach not only improves performance but also opens new avenues for future multi-model collaboration, making it a noteworthy development in the field.