Large reasoning models (LRMs) demonstrate strong reasoning abilities but remain prone to unsafe outputs. Group Relative Policy Optimization (GRPO) is widely adopted for safety alignment, yet its reliance on group‑relative advantage signals often provides insufficient learning guidance. We identify two failure modes:
- Advantage collapse – homogeneous rollout groups produce near‑identical behaviors, causing advantages to vanish and safety signals to disappear.
- Unsafe diversification – when relative signals fade, entropy regularization encourages a broader variety of unsafe responses.
Our theoretical analysis shows that behavioral consistency within groups weakens safety contrasts, while entropy regularization amplifies unsafe diversity under vanishing advantages. To tackle these issues we introduce Beacon, comprising Safety Signal Recovery (SSR) and Safety‑Utility Decoupled Alignment (SUDA).
- SSR recovers effective safety signals by filtering trajectories with advantage guidance and supervising against safe reference outputs.
- SUDA applies category‑aware rewards and a benign‑only KL anchoring to avoid over‑refusal without sacrificing safety.
Experiments on four Qwen3 and DeepSeek‑R1‑Distill variants across eight benchmarks demonstrate that Beacon achieves the best trade‑off between safety and reasoning, reducing average safety error by up to 32.83 percentage points while keeping over‑refusal low.
Review: Beacon’s signal recovery and decoupling mechanisms successfully address GRPO’s core shortcomings, offering a practical route for safe alignment of large‑scale reasoning models.