Policy Gradient from Demonstration and Curiosity
This work addresses the problem of sparse rewards in reinforcement learning for agents, offering a solution that reduces reliance on extensive expert demonstrations, though it is incremental in nature.
The paper tackled the challenge of sparse extrinsic rewards in reinforcement learning by proposing an integrated policy gradient algorithm that uses limited expert demonstrations and intrinsic curiosity to boost exploration. The method achieved superior exploration efficiency and high average return across simulated tasks with only a single demonstration per task.
With reinforcement learning, an agent could learn complex behaviors from high-level abstractions of the task. However, exploration and reward shaping remained challenging for existing methods, especially in scenarios where the extrinsic feedback was sparse. Expert demonstrations have been investigated to solve these difficulties, but a tremendous number of high-quality demonstrations were usually required. In this work, an integrated policy gradient algorithm was proposed to boost exploration and facilitate intrinsic reward learning from only limited number of demonstrations. We achieved this by reformulating the original reward function with two additional terms, where the first term measured the Jensen-Shannon divergence between current policy and the expert, and the second term estimated the agent's uncertainty about the environment. The presented algorithm was evaluated on a range of simulated tasks with sparse extrinsic reward signals where only one single demonstrated trajectory was provided to each task, superior exploration efficiency and high average return were demonstrated in all tasks. Furthermore, it was found that the agent could imitate the expert's behavior and meanwhile sustain high return.