On Diversified Preferences of Large Language Model Alignment
This work addresses the challenge of aligning LLMs with varied human preferences, which is crucial for improving interaction quality in real-world applications, but it is incremental as it builds on existing alignment methods.
This paper tackles the problem of aligning large language models with diverse human preferences by analyzing how model and data size affect this alignment, revealing that larger models mitigate negative effects while smaller ones struggle. It introduces a new metric, Expected Calibration Error, and a method called MORE to improve reward model calibration, achieving superior alignment performance in experiments across four models and five datasets.
Aligning large language models (LLMs) with human preferences has been recognized as the key to improving LLMs' interaction quality. However, in this pluralistic world, human preferences can be diversified due to annotators' different tastes, which hinders the effectiveness of LLM alignment methods. This paper presents the first quantitative analysis of the experimental scaling law for reward models with varying sizes, from 1.3 billion to 7 billion parameters, trained with human feedback exhibiting diverse preferences. Our analysis reveals that the impact of diversified human preferences depends on both model size and data size. Larger models with sufficient capacity mitigate the negative effects of diverse preferences, while smaller models struggle to accommodate them. To mitigate the impact of diverse preferences, we introduce a new metric, Expected Calibration Error (ECE), to evaluate RMs and show their obvious positive correlation with the alignment performance of LLMs. Furthermore, we propose a Multi-Objective Reward learning method (MORE) to enhance the calibration performance of RMs on shared preferences. Through experiments on four models and five human preference datasets, we find the calibration error can be adopted as a key metric for evaluating RMs and MORE can obtain superior alignment performance.