Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment
This work addresses the challenge of efficient human alignment for LLMs, offering a data allocation strategy that could reduce annotation costs, though it appears incremental in refining existing fine-tuning approaches.
The study tackled the problem of aligning large language models with human preferences under limited annotation resources by comparing the impact of diversifying prompts versus responses. It found that using more diverse responses with fewer prompts is more effective for alignment, and proposed a new formulation for prompt diversity that linearly correlates with model performance.
Alignment with human preference prevents large language models (LLMs) from generating misleading or toxic content while requiring high-cost human feedback. Assuming resources of human annotation are limited, there are two different ways of allocating considered: more diverse PROMPTS or more diverse RESPONSES to be labeled. Nonetheless, a straightforward comparison between their impact is absent. In this work, we first control the diversity of both sides according to the number of samples for fine-tuning, which can directly reflect their influence. We find that instead of numerous prompts, more responses but fewer prompts better trigger LLMs for human alignment. Additionally, the concept of diversity for prompts can be more complex than responses that are typically quantified by single digits. Consequently, a new formulation of prompt diversity is proposed, further implying a linear correlation with the final performance of LLMs after fine-tuning. We also leverage it on data augmentation and conduct experiments to show its effect on different algorithms.