Diverse Preference Learning for Capabilities and Alignment
This addresses a critical issue for AI safety and fairness by enhancing LLM diversity, though it is incremental as it builds on existing preference learning methods.
The paper tackles the problem of reduced diversity in LLM outputs caused by alignment algorithms like RLHF and DPO, proposing Soft Preference Learning to decouple KL divergence terms for better control, resulting in higher accuracy on repeated sampling tasks and improved representation of societal viewpoints.
The ability of LLMs to represent diverse perspectives is critical as they increasingly impact society. However, recent studies reveal that alignment algorithms such as RLHF and DPO significantly reduce the diversity of LLM outputs. Not only do aligned LLMs generate text with repetitive structure and word choice, they also approach problems in more uniform ways, and their responses reflect a narrower range of societal perspectives. We attribute this problem to the KL divergence regularizer employed in preference learning algorithms. This causes the model to systematically overweight majority opinions and sacrifice diversity in its outputs. To address this, we propose Soft Preference Learning, which decouples the entropy and cross-entropy terms in the KL penalty - allowing for fine-grained control over LLM generation diversity. From a capabilities perspective, LLMs trained using Soft Preference Learning attain higher accuracy on difficult repeated sampling tasks and produce outputs with greater semantic and lexical diversity. From an alignment perspective, they are capable of representing a wider range of societal viewpoints and display improved logit calibration. Notably, Soft Preference Learning resembles, but is a Pareto improvement over, standard temperature scaling.