Data Driven Reward Initialization for Preference based Reinforcement Learning
This work addresses a specific technical bottleneck in PbRL for researchers and practitioners, making the method more reliable and efficient, but it is incremental as it builds on existing PbRL frameworks.
The paper tackles the problem of high variability in initialized reward models in Preference-based Reinforcement Learning (PbRL) due to random seeds, which exacerbates existing issues with degenerate reward functions. The result is a data-driven reward initialization method that reduces performance variability across runs and improves overall performance compared to other methods, without adding significant cost to the human or agent.
Preference-based Reinforcement Learning (PbRL) methods utilize binary feedback from the human in the loop (HiL) over queried trajectory pairs to learn a reward model in an attempt to approximate the human's underlying reward function capturing their preferences. In this work, we investigate the issue of a high degree of variability in the initialized reward models which are sensitive to random seeds of the experiment. This further compounds the issue of degenerate reward functions PbRL methods already suffer from. We propose a data-driven reward initialization method that does not add any additional cost to the human in the loop and negligible cost to the PbRL agent and show that doing so ensures that the predicted rewards of the initialized reward model are uniform in the state space and this reduces the variability in the performance of the method across multiple runs and is shown to improve the overall performance compared to other initialization methods.