Hindsight Experience Replay
This addresses the problem of sparse rewards for reinforcement learning practitioners, particularly in robotics, by providing a method that simplifies training without requiring reward engineering, though it is incremental as it builds on existing off-policy algorithms.
The paper tackles the challenge of sparse rewards in reinforcement learning by introducing Hindsight Experience Replay, which enables sample-efficient learning with binary rewards and eliminates the need for complex reward engineering, as demonstrated in robotic manipulation tasks like pushing, sliding, and pick-and-place where it made training feasible.
Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are sparse and binary and therefore avoid the need for complicated reward engineering. It can be combined with an arbitrary off-policy RL algorithm and may be seen as a form of implicit curriculum. We demonstrate our approach on the task of manipulating objects with a robotic arm. In particular, we run experiments on three different tasks: pushing, sliding, and pick-and-place, in each case using only binary rewards indicating whether or not the task is completed. Our ablation studies show that Hindsight Experience Replay is a crucial ingredient which makes training possible in these challenging environments. We show that our policies trained on a physics simulation can be deployed on a physical robot and successfully complete the task.