Multi-Task Reward Learning from Human Ratings
This addresses the challenge of aligning model behavior with user goals in RLHF by better capturing human reasoning, though it appears incremental as it builds on existing RLHF frameworks.
The paper tackles the problem of reinforcement learning from human feedback (RLHF) by proposing a method that jointly considers multiple tasks to mimic human decision-making, using synthetic human ratings to infer a reward function with learnable weights, and results show it outperforms existing rating-based RL methods and sometimes traditional RL approaches.
Reinforcement learning from human feedback (RLHF) has become a key factor in aligning model behavior with users' goals. However, while humans integrate multiple strategies when making decisions, current RLHF approaches often simplify this process by modeling human reasoning through isolated tasks such as classification or regression. In this paper, we propose a novel reinforcement learning (RL) method that mimics human decision-making by jointly considering multiple tasks. Specifically, we leverage human ratings in reward-free environments to infer a reward function, introducing learnable weights that balance the contributions of both classification and regression models. This design captures the inherent uncertainty in human decision-making and allows the model to adaptively emphasize different strategies. We conduct several experiments using synthetic human ratings to validate the effectiveness of the proposed approach. Results show that our method consistently outperforms existing rating-based RL methods, and in some cases, even surpasses traditional RL approaches.