Large language models (LLMs) undergo safety alignment but remain vulnerable to jailbreak attacks. Recent studies reveal that alignment is often highly sparse: pruning a few safety‑critical neurons or removing a single refusal direction can drastically weaken aligned behavior. To reduce reliance on such sparse components, we introduce Addition by Subtraction. During finetuning we gradually eliminate identified safety‑critical neurons and employ preference optimization to re‑establish safety‑aligned behavior in other parts of the model. Results show that models trained with this approach are markedly more robust to pruning‑based attacks. Importantly, the robustness generalizes to other sparsity interventions (e.g., refusal‑direction ablation) and a wide range of black‑box jailbreak attacks. The finetuned models also exhibit improved truthfulness while largely preserving overall capabilities. Comprehensive experiments on four models from three vendors demonstrate consistent safety gains, with attack success rates dropping up to 90%.
Review: This study provides a practical pathway to strengthen LLM safety by systematically removing vulnerable neurons and redistributing safety functions, offering a promising direction for more resilient alignment.