Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedback
This work addresses the problem of aligning large language models with diverse human preferences, offering an incremental improvement over existing RLHF methods by incorporating context and handling distribution shifts.
The paper tackles the challenge of heterogeneous human feedback in reinforcement learning from human feedback (RLHF) by proposing a Low-rank Contextual RLHF framework, which integrates contextual information and uses a pessimistic policy to achieve a tighter sub-optimality gap and superior performance in personalized settings.
Reinforcement learning from human feedback (RLHF) has become a cornerstone for aligning large language models with human preferences. However, the heterogeneity of human feedback, driven by diverse individual contexts and preferences, poses significant challenges for reward learning. To address this, we propose a Low-rank Contextual RLHF (LoCo-RLHF) framework that integrates contextual information to better model heterogeneous feedback while maintaining computational efficiency. Our approach builds on a contextual preference model, leveraging the intrinsic low-rank structure of the interaction between user contexts and query-answer pairs to mitigate the high dimensionality of feature representations. Furthermore, we address the challenge of distributional shifts in feedback through our Pessimism in Reduced Subspace (PRS) policy, inspired by pessimistic offline reinforcement learning techniques. We theoretically demonstrate that our policy achieves a tighter sub-optimality gap compared to existing methods. Extensive experiments validate the effectiveness of LoCo-RLHF, showcasing its superior performance in personalized RLHF settings and its robustness to distribution shifts.