Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF
This work addresses vulnerabilities in RLHF deployment for AI systems like chatbots, offering a method to mitigate issues from hidden context, though it is incremental in improving existing preference learning frameworks.
The paper tackles the problem of hidden context in preference learning from human feedback, showing that standard methods like RLHF implicitly aggregate preferences via Borda count, which can lead to counter-intuitive results and vulnerabilities such as jailbreak attacks. It introduces distributional preference learning (DPL) methods, which estimate score distributions to account for hidden context, and experiments show that applying DPL to RLHF for LLM chatbots significantly reduces jailbreak vulnerability.
In practice, preference learning from human feedback depends on incomplete data with hidden context. Hidden context refers to data that affects the feedback received, but which is not represented in the data used to train a preference model. This captures common issues of data collection, such as having human annotators with varied preferences, cognitive processes that result in seemingly irrational behavior, and combining data labeled according to different criteria. We prove that standard applications of preference learning, including reinforcement learning from human feedback (RLHF), implicitly aggregate over hidden contexts according to a well-known voting rule called Borda count. We show this can produce counter-intuitive results that are very different from other methods which implicitly aggregate via expected utility. Furthermore, our analysis formalizes the way that preference learning from users with diverse values tacitly implements a social choice function. A key implication of this result is that annotators have an incentive to misreport their preferences in order to influence the learned model, leading to vulnerabilities in the deployment of RLHF. As a step towards mitigating these problems, we introduce a class of methods called distributional preference learning (DPL). DPL methods estimate a distribution of possible score values for each alternative in order to better account for hidden context. Experimental results indicate that applying DPL to RLHF for LLM chatbots identifies hidden context in the data and significantly reduces subsequent jailbreak vulnerability. Our code and data are available at https://github.com/cassidylaidlaw/hidden-context