GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning
This addresses a critical safety issue for users of fine-tuned LLMs, though it is an incremental improvement over existing methods like joint optimization with original data.
The paper tackles the problem of preserving safety alignment in large language models during fine-tuning, which is often compromised, by proposing GR-SAP, a framework that synthesizes domain-specific alignment data to mitigate safety degradation while maintaining downstream performance.
Recent studies show that the safety alignment of large language models (LLMs) can be easily compromised even by seemingly non-adversarial fine-tuning. To preserve safety alignment during fine-tuning, a widely used strategy is to jointly optimize safety and task objectives by mixing in the original alignment data, which is typically inaccessible even for open-weight LLMs. Inspired by generative replay in continual learning, we propose Generative Replay for Safety Alignment Preservation (GR-SAP), a unified framework that synthesizes domain-specific alignment data from LLMs and integrate them during downstream adaption to preserve safety alignment. Theoretical and empirical analyses demonstrate this synthetic data serves as a reliable proxy for the original alignment data. Experiments across various models and downstream tasks show that GR-SAP substantially mitigates fine-tuning-induced safety degradation while maintaining comparable downstream performance. Our code is available at https://github.com/chili-lab/gr-sap.