Improving LLM Safety Alignment with Dual-Objective Optimization
This work addresses safety alignment for LLMs, which is crucial for preventing harmful outputs in real-world applications, though it appears incremental as it builds upon existing DPO methods.
The paper tackled the vulnerability of large language models (LLMs) to jailbreak attacks by proposing an improved safety alignment method that disentangles direct preference optimization (DPO) objectives into robust refusal training and targeted unlearning of harmful knowledge, resulting in significantly increased robustness against a wide range of jailbreak attacks across in-distribution and out-of-distribution scenarios.
Existing training-time safety alignment techniques for large language models (LLMs) remain vulnerable to jailbreak attacks. Direct preference optimization (DPO), a widely deployed alignment method, exhibits limitations in both experimental and theoretical contexts as its loss function proves suboptimal for refusal learning. Through gradient-based analysis, we identify these shortcomings and propose an improved safety alignment that disentangles DPO objectives into two components: (1) robust refusal training, which encourages refusal even when partial unsafe generations are produced, and (2) targeted unlearning of harmful knowledge. This approach significantly increases LLM robustness against a wide range of jailbreak attacks, including prefilling, suffix, and multi-turn attacks across both in-distribution and out-of-distribution scenarios. Furthermore, we introduce a method to emphasize critical refusal tokens by incorporating a reward-based token-level weighting mechanism for refusal learning, which further improves the robustness against adversarial exploits. Our research also suggests that robustness to jailbreak attacks is correlated with token distribution shifts in the training process and internal representations of refusal and harmful tokens, offering valuable directions for future research in LLM safety alignment. The code is available at https://github.com/wicai24/DOOR-Alignment