In the realm of Reinforcement Learning (RL), despite achieving remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs), existing critic-free RL methods depend on uniform credit assignment, which broadcasts the same advantage across all tokens, disregarding their differences.
We identify a critical failure mode of this design, termed Positive-Credit Contamination: low-probability tail tokens that are contextually erroneous receive identical positive credit to plausible ones within the same trajectory, leading to indiscriminate reinforcement of flawed reasoning behavior.
To address this issue, we propose Tail-Aware Credit Calibration (TACO), a method that calibrates uniform credit assignment to suppress undesirable positive updates.
TACO first computes a tail-risk score that incorporates local generation context to assess each token's risk of falling into the unreliable tail, distinguishing unexpected rarity from uncertainty-driven exploration.
It then uses this score to tune positive credit for risky tokens without completely removing their gradients, allowing recurring useful rare patterns to accumulate reinforcement while progressively dampening incidental noise.
Experimental results across three LLMs and eight benchmarks demonstrate that TACO consistently outperforms GRPO-style baselines. Notably, TACO improves training stability, supporting sustained performance gains in long-horizon RL.
The source code is available at: GitHub.
Blogger's Review: The TACO method emphasizes the significance of fine-tuning risk scores to handle low-probability tail tokens in reinforcement learning. This innovation not only enhances the model's reasoning capabilities but also provides new insights for future RL research, making it a noteworthy area for further exploration.