When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning
This addresses the need for better evaluation in personalized preference learning for LLMs, which is incremental as it focuses on assessment rather than new methods.
The paper tackles the problem of evaluating personalized preference learning in LLMs, which lacks standardized methods, by introducing a multi-faceted framework that measures performance, fairness, unintended effects, and adaptability; results show performance differences up to 36% and safety misalignments up to 20% across methods.
While Reinforcement Learning from Human Feedback (RLHF) is widely used to align Large Language Models (LLMs) with human preferences, it typically assumes homogeneous preferences across users, overlooking diverse human values and minority viewpoints. Although personalized preference learning addresses this by tailoring separate preferences for individual users, the field lacks standardized methods to assess its effectiveness. We present a multi-faceted evaluation framework that measures not only performance but also fairness, unintended effects, and adaptability across varying levels of preference divergence. Through extensive experiments comparing eight personalization methods across three preference datasets, we demonstrate that performance differences between methods could reach 36% when users strongly disagree, and personalization can introduce up to 20% safety misalignment. These findings highlight the critical need for holistic evaluation approaches to advance the development of more effective and inclusive preference learning systems.