Steerable Pluralism: Pluralistic Alignment via Few-Shot Comparative Regression
This work addresses the problem of aligning LLMs with diverse user preferences rather than average preferences, which is important for making AI more fair and representative, though it appears incremental in advancing pluralistic alignment.
The paper tackles the limitation of current LLM alignment methods that use scalar rewards by proposing a steerable pluralistic model based on few-shot comparative regression to capture diverse user preferences across attributes. The approach outperforms baseline and state-of-the-art methods on adapted benchmarks, demonstrating applicability to value-aligned decision-making and reward modeling.
Large language models (LLMs) are currently aligned using techniques such as reinforcement learning from human feedback (RLHF). However, these methods use scalar rewards that can only reflect user preferences on average. Pluralistic alignment instead seeks to capture diverse user preferences across a set of attributes, moving beyond just helpfulness and harmlessness. Toward this end, we propose a steerable pluralistic model based on few-shot comparative regression that can adapt to individual user preferences. Our approach leverages in-context learning and reasoning, grounded in a set of fine-grained attributes, to compare response options and make aligned choices. To evaluate our algorithm, we also propose two new steerable pluralistic benchmarks by adapting the Moral Integrity Corpus (MIC) and the HelpSteer2 datasets, demonstrating the applicability of our approach to value-aligned decision-making and reward modeling, respectively. Our few-shot comparative regression approach is interpretable and compatible with different attributes and LLMs, while outperforming multiple baseline and state-of-the-art methods. Our work provides new insights and research directions in pluralistic alignment, enabling a more fair and representative use of LLMs and advancing the state-of-the-art in ethical AI.