In the real world, agents (e.g., making phone calls) must learn online from costly and often irreversible interactions rather than cheap simulator steps. This leads to two key points:
First, deployability depends on the path, not just the outcome. The agent must adhere to outcome-neutral constraints, such as not repeatedly calling unresponsive users, respecting business hours, or completing necessary authentication constraints, since frequent violations can superficially enhance success rates.
Second, since each interaction is expensive, the agent must learn efficiently from very few examples. Traditional Reinforcement Learning from Verifiable Rewards (RLVR) ignores both challenges, optimizing solely on the outcome and wasting costly rollouts on all-fail groups where group-relative advantage collapses to zero.
Attempts to densify supervision by rewarding progress target the hard-to-verify direction. In contrast, real agentic environments can cheaply detect bad moves. Since group-relative advantage is equivalent to within-group variance, a dense signal helps only when it supplies variance that the outcome lacks.
A verifiable penalty on the path meets this condition reliably, while progress potential helps only when partial progress is achievable. The resulting strategy, "penalize the path, reward the outcome," achieves high task success with near-zero violations, where outcome-only training violates constraints in nearly every episode.
We provide four design rules for effective penalties, including avoiding the inaction trap that arises when a penalty is used in isolation.
Blogger's Review: This article presents a novel approach in reinforcement learning by shifting focus from outcomes to path constraints, which significantly enhances the learning efficiency and deployability of agents in high-cost real-world scenarios. The proposed design rules also offer practical guidance for future research in this domain.