ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference
This work addresses the problem of aligning language models to be helpful and harmless for AI safety and deployment, but it is incremental as it builds on existing preference learning methods like RLHF and DPO.
The paper tackles the challenge of aligning language models to human expectations by addressing the inadequacy of pairwise preference methods for point-wise feedback, introducing Point-wise Direct Preference Optimization and a unified single-step method that combines human demonstrations and point-wise preferences, with experiments validating effectiveness on datasets with binary or continuous labels.
Aligning language models to human expectations, e.g., being helpful and harmless, has become a pressing challenge for large language models. A typical alignment procedure consists of supervised fine-tuning and preference learning. Most preference learning methods, such as RLHF and DPO, depend on pairwise preference data, which inadequately address scenarios where human feedback is point-wise, leading to potential information loss and suboptimal performance. Addressing this gap, we introduce Point-wise Direct Preference Optimization, a novel preference learning method designed to harness point-wise feedback effectively. Our work also uncovers a novel connection between supervised fine-tuning and point-wise preference learning, culminating in Unified Language Model Alignment, a single-step method that unifies the alignment with human demonstrations and point-wise preferences. Extensive experiments on point-wise preference datasets with binary or continuous labels validate the effectiveness of our methods. Our code and a new dataset with high-quality demonstration samples on harmlessness are released.