Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning
This work addresses the challenge of feedback efficiency for researchers and practitioners in reinforcement learning, though it appears incremental as it builds on existing human-in-the-loop methods.
The paper tackled the problem of reducing human interactions in human-in-the-loop reinforcement learning by introducing Sub-optimal Data Pre-training (SDP), which leverages reward-free, sub-optimal data to pre-train reward models, resulting in performance that meets or significantly improves state-of-the-art methods across simulated robotic tasks.
To create useful reinforcement learning (RL) agents, step zero is to design a suitable reward function that captures the nuances of the task. However, reward engineering can be a difficult and time-consuming process. Instead, human-in-the-loop RL methods hold the promise of learning reward functions from human feedback. Despite recent successes, many of the human-in-the-loop RL methods still require numerous human interactions to learn successful reward functions. To improve the feedback efficiency of human-in-the-loop RL methods (i.e., require less human interaction), this paper introduces Sub-optimal Data Pre-training, SDP, an approach that leverages reward-free, sub-optimal data to improve scalar- and preference-based RL algorithms. In SDP, we start by pseudo-labeling all low-quality data with the minimum environment reward. Through this process, we obtain reward labels to pre-train our reward model without requiring human labeling or preferences. This pre-training phase provides the reward model a head start in learning, enabling it to recognize that low-quality transitions should be assigned low rewards. Through extensive experiments with both simulated and human teachers, we find that SDP can at least meet, but often significantly improve, state of the art human-in-the-loop RL performance across a variety of simulated robotic tasks.