Towards Low-Resource Alignment to Diverse Perspectives with Sparse Feedback
This addresses the issue of generic responses in language models for high-stakes applications, though it is incremental as it builds on existing alignment methods.
The paper tackled the problem of aligning language models to diverse perspectives in low-resource settings, showing that model steering improves over baselines with only 50 annotated samples and reduces false positives in tasks like hate speech detection.
As language models have a greater impact on society, it is important to ensure they are aligned to a diverse range of perspectives and are able to reflect nuance in human values. However, the most popular training paradigms for modern language models often assume there is one optimal answer for every query, leading to generic responses and poor alignment. In this work, we aim to enhance pluralistic alignment of language models in a low-resource setting with two methods: pluralistic decoding and model steering. We empirically demonstrate that model steering offers consistent improvement over zero-shot and few-shot baselines with only 50 annotated samples. Our proposed methods decrease false positives in several high-stakes tasks such as hate speech detection and misinformation detection, and improves the distributional alignment to human values in GlobalOpinionQA. We hope our work highlights the importance of diversity and how language models can be adapted to consider nuanced perspectives.