Multi-Response Preference Optimization with Augmented Ranking Dataset
This work addresses dataset quality issues in preference optimization for LLMs, which is an incremental improvement in training methods.
The paper tackles the challenge of constructing high-quality preference optimization datasets for LLMs by proposing a novel dataset augmentation approach and a multi-response-based training method that learns from multiple responses simultaneously.
Recent advancements in Large Language Models (LLMs) have been remarkable, with new models consistently surpassing their predecessors. These advancements are underpinned by extensive research on various training mechanisms. Among these, Preference Optimization has played a significant role in improving the performance of LLMs by incorporating human preferences into the training process. However, constructing preference optimization datasets is challenging and the optimization process is highly sensitive to the dataset quality. In this study, we propose a novel approach to augment Preference Optimization datasets. Additionally, we introduce a Multi-response-based Preference Optimization training method that enables the simultaneous learning of multiple responses.