Human Preferences for Constructive Interactions in Language Model Alignment
This research addresses the problem of aligning LLMs to promote constructive dialogue and reduce societal divisions, though it is incremental as it builds on existing preference-based alignment methods.
The study analyzed human preferences in LLM interactions using a multicultural dataset of 7,500 conversations, finding that users generally preferred well-reasoned responses over personal storytelling, but those valuing AI alignment to their own values prioritized curiosity over reasoning, and LLMs mirrored user attributes like toxicity.
As large language models (LLMs) enter the mainstream, aligning them to foster constructive dialogue rather than exacerbate societal divisions is critical. Using an individualized and multicultural alignment dataset of over 7,500 conversations of individuals from 74 countries engaging with 21 LLMs, we examined how linguistic attributes linked to constructive interactions are reflected in human preference data used for training AI. We found that users consistently preferred well-reasoned and nuanced responses while rejecting those high in personal storytelling. However, users who believed that AI should reflect their values tended to place less preference on reasoning in LLM responses and more on curiosity. Encouragingly, we observed that users could set the tone for how constructive their conversation would be, as LLMs mirrored linguistic attributes, including toxicity, in user queries.