FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions
This work addresses sample-efficient adaptation of pre-trained robots to human preferences, which is incremental as it builds on existing PbRL methods to mitigate a specific bottleneck.
The paper tackles the problem of catastrophic reward forgetting in preference-based reinforcement learning for robotic style adaptation, where existing methods overfit to new preferences and fail at the original task. The result is a method using low-rank matrices to adapt reward functions, which efficiently adjusts robotic behavior to human preferences in simulation and real-world tasks.
Preference-based reinforcement learning (PbRL) is a suitable approach for style adaptation of pre-trained robotic behavior: adapting the robot's policy to follow human user preferences while still being able to perform the original task. However, collecting preferences for the adaptation process in robotics is often challenging and time-consuming. In this work we explore the adaptation of pre-trained robots in the low-preference-data regime. We show that, in this regime, recent adaptation approaches suffer from catastrophic reward forgetting (CRF), where the updated reward model overfits to the new preferences, leading the agent to become unable to perform the original task. To mitigate CRF, we propose to enhance the original reward model with a small number of parameters (low-rank matrices) responsible for modeling the preference adaptation. Our evaluation shows that our method can efficiently and effectively adjust robotic behavior to human preferences across simulation benchmark tasks and multiple real-world robotic tasks.