Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models
This work addresses the challenge of sample efficiency and safety in preference-based reinforcement learning for robotics, representing an incremental improvement over prior approaches.
The paper tackles the problem of enabling robots to learn tasks based on user preferences without hand-crafted rewards, showing that using a learned dynamics model reduces environment interactions by 50% compared to model-free methods and allows safe, efficient preference query synthesis.
Preference-based reinforcement learning (PbRL) can enable robots to learn to perform tasks based on an individual's preferences without requiring a hand-crafted reward function. However, existing approaches either assume access to a high-fidelity simulator or analytic model or take a model-free approach that requires extensive, possibly unsafe online environment interactions. In this paper, we study the benefits and challenges of using a learned dynamics model when performing PbRL. In particular, we provide evidence that a learned dynamics model offers the following benefits when performing PbRL: (1) preference elicitation and policy optimization require significantly fewer environment interactions than model-free PbRL, (2) diverse preference queries can be synthesized safely and efficiently as a byproduct of standard model-based RL, and (3) reward pre-training based on suboptimal demonstrations can be performed without any environmental interaction. Our paper provides empirical evidence that learned dynamics models enable robots to learn customized policies based on user preferences in ways that are safer and more sample efficient than prior preference learning approaches. Supplementary materials and code are available at https://sites.google.com/berkeley.edu/mop-rl.