Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards
This work addresses the problem of scalable robotic manipulation skill learning for researchers and practitioners, representing an incremental improvement over existing methods.
The paper tackles the challenge of exploration inefficiency and convergence issues in multi-goal reinforcement learning with sparse rewards by proposing a density-based curriculum learning method, which outperforms state-of-the-art baselines in data efficiency and success rate on benchmark manipulation tasks.
Multi-goal reinforcement learning (RL) aims to qualify the agent to accomplish multi-goal tasks, which is of great importance in learning scalable robotic manipulation skills. However, reward engineering always requires strenuous efforts in multi-goal RL. Moreover, it will introduce inevitable bias causing the suboptimality of the final policy. The sparse reward provides a simple yet efficient way to overcome such limits. Nevertheless, it harms the exploration efficiency and even hinders the policy from convergence. In this paper, we propose a density-based curriculum learning method for efficient exploration with sparse rewards and better generalization to desired goal distribution. Intuitively, our method encourages the robot to gradually broaden the frontier of its ability along the directions to cover the entire desired goal space as much and quickly as possible. To further improve data efficiency and generality, we augment the goals and transitions within the allowed region during training. Finally, We evaluate our method on diversified variants of benchmark manipulation tasks that are challenging for existing methods. Empirical results show that our method outperforms the state-of-the-art baselines in terms of both data efficiency and success rate.