Low‑precision execution can dramatically speed up reinforcement learning (RL) for large language models, yet execution mismatches between the learner and the sampler often destabilize policy optimization. This work first characterizes the interaction between mismatch and the policy‑gradient direction, distinguishing locally amplifying from contracting update contributions—something magnitude alone cannot reveal.
In native NVFP4 (W4A4) runs we observe an early imbalance between the two amplifying regions, dominated by negative‑advantage and negative‑gap updates. Consequently, tail tokens become highly concentrated in a small fraction of response segments before the mismatch spreads globally.
Motivated by these observations, we introduce TRIAGE (Direction‑Aware Stabilization), whose key components are:
- Segment‑level diagnosis: Detect the relative alignment of gradient direction and mismatch at each generated segment, identifying amplifying vs. contracting updates.
- Selective rebalancing: Down‑weight negative updates in amplifying regions and up‑weight positive updates in contracting regions to achieve directional balance.
- Bounded repair: Apply a capped correction to residual severe mismatches, preventing gradient explosion.
TRIAGE modifies the optimization objective to incorporate these mechanisms while retaining native NVFP4 weight‑and‑activation 4‑bit (W4A4) forward execution on both sampler and learner—no fallback to higher precision is required.
Experiments on Qwen3‑4B and Qwen3‑30B‑A3B across five mathematical reasoning benchmarks show:
- Stable optimization throughout the training horizon;
- Full‑precision‑level performance on all benchmarks;
- Up to 2.3× higher rollout throughput compared to BF16 when using native NVFP4 with TRIAGE.
Review: TRIAGE successfully mitigates direction‑aware mismatches in low‑precision RL, delivering both speed and accuracy, and represents a practical solution for large‑model training.