Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
This addresses safety and utility issues in aligned LLMs for users and developers, representing an incremental improvement over existing methods.
The paper tackles the problem of fragile and imbalanced safety mechanisms in aligned large language models, which can lead to unsafe responses, over-safety, and utility loss, by proposing SafePatching, a post safety alignment method that enhances safety, mitigates over-safety, and preserves utility, achieving a more comprehensive alignment than baselines on models like LLaMA-2/3, Gemma, and Mistral.
Safety alignment of large language models (LLMs) has been gaining increasing attention. However, current safety-aligned LLMs suffer from the fragile and imbalanced safety mechanisms, which can still be induced to generate unsafe responses, exhibit over-safety by rejecting safe user inputs, and fail to preserve general utility after safety alignment. To this end, we propose a novel post safety alignment (PSA) method to address these inherent and emerging safety challenges, including safety enhancement, over-safety mitigation, and utility preservation. In specific, we introduce \textsc{SafePatching}, a novel framework for comprehensive PSA, where two distinct safety patches are developed on the harmful data to enhance safety and mitigate over-safety concerns, and then seamlessly integrated into the target LLM backbone without compromising its utility. Extensive experiments on four representative aligned LLMs, including LLaMA-2/3, Gemma and Mistral, show that \textsc{SafePatching} achieves a more comprehensive PSA than baseline methods, further optimizing the balance between being helpful and harmless in current aligned LLMs. Also, \textsc{SafePatching} demonstrates its superiority in continual PSA scenarios.