Exploring and Addressing Reward Confusion in Offline Preference Learning
This addresses a critical issue in offline RLHF for AI alignment, though it appears incremental as it builds on existing preference learning frameworks.
The paper tackles the problem of reward confusion in offline Reinforcement Learning from Human Feedback (RLHF) caused by spurious correlations in training data, and proposes a method that significantly reduces this confusion by leveraging preference transitivity and active learning to build a global preference chain.
Spurious correlations in a reward model's training data can prevent Reinforcement Learning from Human Feedback (RLHF) from identifying the desired goal and induce unwanted behaviors. This paper shows that offline RLHF is susceptible to reward confusion, especially in the presence of spurious correlations in offline data. We create a benchmark to study this problem and propose a method that can significantly reduce reward confusion by leveraging transitivity of preferences while building a global preference chain with active learning.