Constructive Large Language Models Alignment with Diverse Feedback
This addresses the issue of suboptimal alignment in LLMs for applications requiring safety and accuracy, though it appears incremental by building on existing feedback methods.
The paper tackles the problem of aligning large language models with human values by introducing Constructive and Diverse Feedback (CDF), a method that combines critique, refinement, and preference feedback tailored to problem difficulty, resulting in superior performance in tasks like question answering, dialog generation, and text summarization while using less training data.
In recent research on large language models (LLMs), there has been a growing emphasis on aligning these models with human values to reduce the impact of harmful content. However, current alignment methods often rely solely on singular forms of human feedback, such as preferences, annotated labels, or natural language critiques, overlooking the potential advantages of combining these feedback types. This limitation leads to suboptimal performance, even when ample training data is available. In this paper, we introduce Constructive and Diverse Feedback (CDF) as a novel method to enhance LLM alignment, inspired by constructivist learning theory. Our approach involves collecting three distinct types of feedback tailored to problems of varying difficulty levels within the training dataset. Specifically, we exploit critique feedback for easy problems, refinement feedback for medium problems, and preference feedback for hard problems. By training our model with this diversified feedback, we achieve enhanced alignment performance while using less training data. To assess the effectiveness of CDF, we evaluate it against previous methods in three downstream tasks: question answering, dialog generation, and text summarization. Experimental results demonstrate that CDF achieves superior performance even with a smaller training dataset.