Residual Reinforcement Learning from Demonstrations
This work addresses the challenge of enabling robots to perform complex tasks without hand-engineered controllers, using demonstrations from non-experts, though it is incremental in building on existing residual RL methods.
The paper tackled the problem of solving high-dimensional robotic manipulation tasks with sparse rewards by extending residual reinforcement learning to learn from visual inputs and demonstrations, achieving better generalization to unseen conditions than behavioral cloning or RL fine-tuning.
Residual reinforcement learning (RL) has been proposed as a way to solve challenging robotic tasks by adapting control actions from a conventional feedback controller to maximize a reward signal. We extend the residual formulation to learn from visual inputs and sparse rewards using demonstrations. Learning from images, proprioceptive inputs and a sparse task-completion reward relaxes the requirement of accessing full state features, such as object and target positions. In addition, replacing the base controller with a policy learned from demonstrations removes the dependency on a hand-engineered controller in favour of a dataset of demonstrations, which can be provided by non-experts. Our experimental evaluation on simulated manipulation tasks on a 6-DoF UR5 arm and a 28-DoF dexterous hand demonstrates that residual RL from demonstrations is able to generalize to unseen environment conditions more flexibly than either behavioral cloning or RL fine-tuning, and is capable of solving high-dimensional, sparse-reward tasks out of reach for RL from scratch.