Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learning
This work addresses the challenge of reducing human feedback requirements in reinforcement learning for robotics, but it is incremental as it builds on existing methods by adding loss functions for unlabeled data.
The paper tackled the problem of improving reward recovery and feedback efficiency in preference-based reinforcement learning by incorporating unlabeled trajectories into the reward learning process, resulting in better performance than the state-of-the-art baseline PEBBLE in locomotion and robotic manipulation tasks.
Preference Based Reinforcement Learning has shown much promise for utilizing human binary feedback on queried trajectory pairs to recover the underlying reward model of the Human in the Loop (HiL). While works have attempted to better utilize the queries made to the human, in this work we make two observations about the unlabeled trajectories collected by the agent and propose two corresponding loss functions that ensure participation of unlabeled trajectories in the reward learning process, and structure the embedding space of the reward model such that it reflects the structure of state space with respect to action distances. We validate the proposed method on one locomotion domain and one robotic manipulation task and compare with the state-of-the-art baseline PEBBLE. We further present an ablation of the proposed loss components across both the domains and find that not only each of the loss components perform better than the baseline, but the synergic combination of the two has much better reward recovery and human feedback sample efficiency.