Extensive Self-Contrast Enables Feedback-Free Language Model Alignment
This addresses the cost barrier for LLM alignment, making it more accessible, though it is incremental as it builds on existing DPO and SFT techniques.
The paper tackles the high cost of human or LLM feedback in language model alignment by introducing Self-Contrast, a feedback-free method that uses self-generated negatives and embedding-based filtering, achieving consistent large-margin improvements over SFT and standard DPO on three datasets, with performance scaling as negatives increase.
Reinforcement learning from human feedback (RLHF) has been a central technique for recent large language model (LLM) alignment. However, its heavy dependence on costly human or LLM-as-Judge preference feedback could stymie its wider applications. In this work, we introduce Self-Contrast, a feedback-free large language model alignment method via exploiting extensive self-generated negatives. With only supervised fine-tuning (SFT) targets, Self-Contrast leverages the LLM itself to generate massive diverse candidates, and harnesses a pre-trained embedding model to filter multiple negatives according to text similarity. Theoretically, we illustrate that in this setting, merely scaling negative responses can still effectively approximate situations with more balanced positive and negative preference annotations. Our experiments with direct preference optimization (DPO) on three datasets show that, Self-Contrast could consistently outperform SFT and standard DPO training by large margins. And as the number of self-generated negatives increases, the performance of Self-Contrast continues to grow. Code and data are available at https://github.com/THUDM/Self-Contrast.