Abstract
Fine-Tuning-as-a-Service (FTaaS) platforms let users train large language models (LLMs) on customized tasks, but this pipeline could erode models' safety alignment. Service providers need to recover models' safety without re-running full alignment or destroying the utility gained from customized tasks.
A line of existing work refers to model parameter merging, which adds a safety patch on the fine-tuned model parameters to shift the model away from unsafe tendencies. However, this merging-based paradigm is fundamentally bottlenecked by task-safety update entanglement: downstream task updates and the safety patch often overlap in their dominant directions, making it intrinsically hard to calibrate the merge strength. If the safety vector is scaled too weakly, harmful components could still dominate, preventing the model from returning to a safe region; if it is scaled too aggressively, it suppresses task-relevant directions and degrades utility.
To solve this problem, we shift the focus of merging-based methods from designing online merging operators to offline patch learning, seeking a safety patch that minimally interferes with task-relevant directions while retaining decisive control over unsafe behaviors. We propose TRACE, a trajectory-based safety patch learning framework that (i) simulates harmful tuning trajectories to generate progressively corrupted states, and (ii) optimizes a plug-in patch to recover safety while maintaining utility across varying corrupted base states. Across six benchmarks and two models, TRACE consistently dominates the safety-utility frontier. TRACE reaches nearly 100% safety on all settings while maintaining comparable utility to the undefended fine-tuned model.
Blogger's Review: TRACE effectively addresses the conflict between model safety and utility through the innovative combination of trajectory simulation and patch learning. Its outstanding performance across multiple benchmarks highlights its significant potential applications in AI safety, warranting attention and further research in the industry.