Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations
This addresses the challenge of high sample complexity and reliance on dense rewards in real-world reinforcement learning applications, though it is incremental as it builds on existing IL and RL methods.
The paper tackles the problem of learning sparse-rewarded tasks with limited sub-optimal demonstrations by proposing Self-Adaptive Imitation Learning (SAIL), which bridges imitation learning and reinforcement learning to achieve near-optimal performance with improved sample efficiency and better final results across continuous control tasks.
Model-free deep reinforcement learning (RL) has demonstrated its superiority on many complex sequential decision-making problems. However, heavy dependence on dense rewards and high sample-complexity impedes the wide adoption of these methods in real-world scenarios. On the other hand, imitation learning (IL) learns effectively in sparse-rewarded tasks by leveraging the existing expert demonstrations. In practice, collecting a sufficient amount of expert demonstrations can be prohibitively expensive, and the quality of demonstrations typically limits the performance of the learning policy. In this work, we propose Self-Adaptive Imitation Learning (SAIL) that can achieve (near) optimal performance given only a limited number of sub-optimal demonstrations for highly challenging sparse reward tasks. SAIL bridges the advantages of IL and RL to reduce the sample complexity substantially, by effectively exploiting sup-optimal demonstrations and efficiently exploring the environment to surpass the demonstrated performance. Extensive empirical results show that not only does SAIL significantly improve the sample-efficiency but also leads to much better final performance across different continuous control tasks, comparing to the state-of-the-art.