SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning
This addresses the feedback-efficiency issue for researchers and practitioners applying preference-based RL to real-world tasks, though it is incremental as it builds on existing semi-supervised and data augmentation techniques.
The paper tackles the data-efficiency problem in preference-based reinforcement learning, which requires large amounts of human feedback, by introducing SURF, a semi-supervised reward learning framework that uses unlabeled samples and data augmentation. The result is a significant improvement in feedback-efficiency over the state-of-the-art method on locomotion and robotic manipulation tasks.
Preference-based reinforcement learning (RL) has shown potential for teaching agents to perform the target tasks without a costly, pre-defined reward function by learning the reward with a supervisor's preference between the two agent behaviors. However, preference-based learning often requires a large amount of human feedback, making it difficult to apply this approach to various applications. This data-efficiency problem, on the other hand, has been typically addressed by using unlabeled samples or data augmentation techniques in the context of supervised learning. Motivated by the recent success of these approaches, we present SURF, a semi-supervised reward learning framework that utilizes a large amount of unlabeled samples with data augmentation. In order to leverage unlabeled samples for reward learning, we infer pseudo-labels of the unlabeled samples based on the confidence of the preference predictor. To further improve the label-efficiency of reward learning, we introduce a new data augmentation that temporally crops consecutive subsequences from the original behaviors. Our experiments demonstrate that our approach significantly improves the feedback-efficiency of the state-of-the-art preference-based method on a variety of locomotion and robotic manipulation tasks.