SDGO: Self-Discrimination-Guided Optimization for Consistent Safety in Large Language Models
This work addresses safety vulnerabilities in LLMs for users and developers, offering a method to align discrimination and generation capabilities without additional data, though it is incremental as it builds on existing reinforcement learning and safety alignment techniques.
The paper tackles the problem of safety inconsistency in Large Language Models (LLMs), where models are better at identifying harmful requests than defending against them as generators, and proposes SDGO, a reinforcement learning framework that uses the model's own discrimination capabilities as a reward signal to enhance safety, achieving significant improvements in safety against jailbreaking attacks while maintaining helpfulness on general benchmarks.
Large Language Models (LLMs) excel at various natural language processing tasks but remain vulnerable to jailbreaking attacks that induce harmful content generation. In this paper, we reveal a critical safety inconsistency: LLMs can more effectively identify harmful requests as discriminators than defend against them as generators. This insight inspires us to explore aligning the model's inherent discrimination and generation capabilities. To this end, we propose SDGO (Self-Discrimination-Guided Optimization), a reinforcement learning framework that leverages the model's own discrimination capabilities as a reward signal to enhance generation safety through iterative self-improvement. Our method does not require any additional annotated data or external models during the training phase. Extensive experiments demonstrate that SDGO significantly improves model safety compared to both prompt-based and training-based baselines while maintaining helpfulness on general benchmarks. By aligning LLMs' discrimination and generation capabilities, SDGO brings robust performance against out-of-distribution (OOD) jailbreaking attacks. This alignment achieves tighter coupling between these two capabilities, enabling the model's generation capability to be further enhanced with only a small amount of discriminative samples. Our code and datasets are available at https://github.com/NJUNLP/SDGO.