Continual Safety Alignment via Gradient-Based Sample Selection
For practitioners fine-tuning LLMs on new domains, this work offers a simple, data-centric method to maintain safety without extra safe data or architectural changes.
Large language models lose safety alignment when fine-tuned on new tasks. The authors show that high-gradient training samples cause this drift and propose filtering them out, which preserves alignment with minimal task performance loss across multiple models and benchmarks.
Large language models require continuous adaptation to new tasks while preserving safety alignment. However, fine-tuning on even benign data often compromises safety behaviors, including refusal of harmful requests, truthfulness, and commonsense reasoning. We investigate which training samples cause alignment drift through a data-centric lens. Our empirical analysis shows samples contribute unequally: high-gradient samples cause greater safety degradation and drive models toward pretrained distributions, while moderate-gradient samples enable task learning with minimal alignment loss. We propose gradient-based sample selection that filters high-gradient samples during fine-tuning. Across multiple model families on continual domain tasks, our method substantially improves alignment preservation while maintaining competitive task performance, without requiring curated safe data or architectural modifications. Our method is robust across selection ratios, task orderings, and diverse attack benchmarks.