NeFut Logo NeFut
中 Admin Login

[CS.AI] SAVeR²: Reasoning‑Based Safety Alignment for Large Reasoning Models

Published at: 2026-11-06 22:00 Last updated: 2026-11-07 12:32
#AI #Machine Learning #optimization

Large Reasoning Models (LRMs) achieve strong performance with explicit chain‑of‑thought (CoT) reasoning, yet this introduces safety hazards. Safety is formalized as three requirements: non‑malicious reasoning and answers, safety consistency, and explicit intent identification. Existing alignment approaches ignore the intrinsic reasoning ability of LRMs, resulting in poor adversarial generalization, a noticeable safety tax on reasoning, and coarse, unfaithful reward signals.\n\nTo address these issues, we propose SAVeR², a safety alignment method that leverages reasoning capability. The safety objective is decomposed into four fine‑grained verifiable reward signals, and a difficulty‑aware data selection strategy is used to stabilize training. The key innovation is a reasoning‑preserving gradient projection mechanism that analytically resolves conflicts between safety and reasoning gradients, effectively eliminating the safety tax.\n\nExtensive experiments on models from 1.5B to 671B parameters show that SAVeR² achieves superior safety performance, markedly lower over‑refusal rates, and retains general reasoning ability across multiple benchmarks.\n\nReview: SAVeR²’s combination of granular rewards and gradient projection offers a verifiable solution that balances safety and reasoning, advancing the alignment of large reasoning models.

Original Source: https://arxiv.org/abs/2611.07482

[h] Back to Home