Exploring Chain-of-Thought Reasoning for Steerable Pluralistic Alignment
This addresses the need for more adaptable AI in applications requiring diverse human perspectives, though it is incremental as it builds on existing CoT and alignment methods.
The paper tackled the problem of enabling large language models to adopt specific perspectives for nuanced tasks by exploring Chain-of-Thought reasoning techniques, finding that Reinforcement Learning with Verifiable Rewards outperformed other methods in performance and sample efficiency on datasets like Value Kaleidoscope and OpinionQA.
Large Language Models (LLMs) are typically trained to reflect a relatively uniform set of values, which limits their applicability to tasks that require understanding of nuanced human perspectives. Recent research has underscored the importance of enabling LLMs to support steerable pluralism -- the capacity to adopt a specific perspective and align generated outputs with it. In this work, we investigate whether Chain-of-Thought (CoT) reasoning techniques can be applied to building steerable pluralistic models. We explore several methods, including CoT prompting, fine-tuning on human-authored CoT, fine-tuning on synthetic explanations, and Reinforcement Learning with Verifiable Rewards (RLVR). We evaluate these approaches using the Value Kaleidoscope and OpinionQA datasets. Among the methods studied, RLVR consistently outperforms others and demonstrates strong training sample efficiency. We further analyze the generated CoT traces with respect to faithfulness and safety.