A State Augmentation based approach to Reinforcement Learning from Human Preferences
This addresses reward specification problems in RL for researchers and practitioners, though it appears to be an incremental improvement over existing preference-based RL methods.
The paper tackles the problem of reward specification and reward hacking in reinforcement learning by proposing a state augmentation technique for preference-based RL that improves reward recovery and policy performance. Their method achieved significant performance improvements over the PEBBLE baseline across three domains including Mountain Car, Quadruped-Walk, and Sweep-Into tasks.
Reinforcement Learning has suffered from poor reward specification, and issues for reward hacking even in simple enough domains. Preference Based Reinforcement Learning attempts to solve the issue by utilizing binary feedbacks on queried trajectory pairs by a human in the loop indicating their preferences about the agent's behavior to learn a reward model. In this work, we present a state augmentation technique that allows the agent's reward model to be robust and follow an invariance consistency that significantly improved performance, i.e. the reward recovery and subsequent return computed using the learned policy over our baseline PEBBLE. We validate our method on three domains, Mountain Car, a locomotion task of Quadruped-Walk, and a robotic manipulation task of Sweep-Into, and find that using the proposed augmentation the agent not only benefits in the overall performance but does so, quite early in the agent's training phase.