RecoveryChaining: Learning Local Recovery Policies for Robust Manipulation
This work addresses robustness in robot manipulation for deployment in real-world environments, though it is incremental as it builds on existing hierarchical reinforcement learning and model-based control methods.
The paper tackles the problem of model-based planners failing due to actuation noise and imperfect models by proposing RecoveryChaining, a hierarchical reinforcement learning method that learns local recovery policies to transition robots back to states where nominal controllers can succeed, achieving significantly more robust policies in multi-step manipulation tasks with sparse rewards and demonstrating successful sim-to-real transfer.
Model-based planners and controllers are commonly used to solve complex manipulation problems as they can efficiently optimize diverse objectives and generalize to long horizon tasks. However, they often fail during deployment due to noisy actuation, partial observability and imperfect models. To enable a robot to recover from such failures, we propose to use hierarchical reinforcement learning to learn a recovery policy. The recovery policy is triggered when a failure is detected based on sensory observations and seeks to take the robot to a state from which it can complete the task using the nominal model-based controllers. Our approach, called RecoveryChaining, uses a hybrid action space, where the model-based controllers are provided as additional \emph{nominal} options which allows the recovery policy to decide how to recover, when to switch to a nominal controller and which controller to switch to even with \emph{sparse rewards}. We evaluate our approach in three multi-step manipulation tasks with sparse rewards, where it learns significantly more robust recovery policies than those learned by baselines. We successfully transfer recovery policies learned in simulation to a physical robot to demonstrate the feasibility of sim-to-real transfer with our method.