Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization
This addresses the challenge of noisy positive samples in alignment for LLMs, offering a more stable and efficient approach for developers and researchers, though it is incremental as it builds on existing alignment methods.
The paper tackles the problem of aligning large language models with human preferences by proposing a method that uses only human-annotated negative samples to reduce harmfulness while preserving helpfulness, achieving comparable generation quality and surpassing baselines in producing less harmful and more informative responses with better training stability and faster convergence.
Large language models (LLMs) have revolutionized the role of AI, yet pose potential social risks. To steer LLMs towards human preference, alignment technologies have been introduced and gained increasing attention. Nevertheless, existing methods heavily rely on high-quality positive-negative training pairs, suffering from noisy positive responses that are barely distinguishable from negative ones. Given recent LLMs' proficiency in generating helpful responses, this work pivots towards a new research question: can we achieve alignment using solely human-annotated negative samples, preserving helpfulness while reducing harmfulness? For this purpose, we propose Distributional Dispreference Optimization (D$^2$O), which maximizes the discrepancy between dispreferred responses and the generated non-negative ones. In this way, D$^2$O effectively eschews harmful information without incorporating noisy positive samples, while avoiding collapse using self-generated responses as anchors. We demonstrate that D$^2$O can be regarded as learning a distributional preference model reflecting human dispreference against negative responses, which is theoretically an upper bound of the instance-level DPO. Extensive experiments manifest that our method achieves comparable generation quality and surpasses the latest strong baselines in producing less harmful and more informative responses with better training stability and faster convergence.