Listwise Reward Estimation for Offline Preference-based Reinforcement Learning
This work addresses the problem of aligning reward functions with human intent in reinforcement learning for researchers and practitioners, though it is incremental as it builds on existing preference-based RL methods by incorporating second-order preferences.
The paper tackles the challenge of designing precise reward functions in reinforcement learning by proposing Listwise Reward Estimation (LiRE), a novel offline preference-based RL method that leverages second-order preference information, and it demonstrates superiority by outperforming state-of-the-art baselines with modest feedback budgets and robustness to feedback noise.
In Reinforcement Learning (RL), designing precise reward functions remains to be a challenge, particularly when aligning with human intent. Preference-based RL (PbRL) was introduced to address this problem by learning reward models from human feedback. However, existing PbRL methods have limitations as they often overlook the second-order preference that indicates the relative strength of preference. In this paper, we propose Listwise Reward Estimation (LiRE), a novel approach for offline PbRL that leverages second-order preference information by constructing a Ranked List of Trajectories (RLT), which can be efficiently built by using the same ternary feedback type as traditional methods. To validate the effectiveness of LiRE, we propose a new offline PbRL dataset that objectively reflects the effect of the estimated rewards. Our extensive experiments on the dataset demonstrate the superiority of LiRE, i.e., outperforming state-of-the-art baselines even with modest feedback budgets and enjoying robustness with respect to the number of feedbacks and feedback noise. Our code is available at https://github.com/chwoong/LiRE