Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemble
This work addresses sample efficiency and robustness issues in reinforcement learning for real-world robotics, offering an incremental improvement over existing learning from demonstrations methods.
The paper tackles the problem of low sample efficiency and performance degradation in deep reinforcement learning for robot control by proposing A-SILfD, a method that uses expert demonstrations to constrain policy improvement and ensemble Q-functions to reduce estimation errors, resulting in significant performance gains in Mujoco tasks after 150,000 steps.
Deep reinforcement learning (DRL) provides a new way to generate robot control policy. However, the process of training control policy requires lengthy exploration, resulting in a low sample efficiency of reinforcement learning (RL) in real-world tasks. Both imitation learning (IL) and learning from demonstrations (LfD) improve the training process by using expert demonstrations, but imperfect expert demonstrations can mislead policy improvement. Offline to Online reinforcement learning requires a lot of offline data to initialize the policy, and distribution shift can easily lead to performance degradation during online fine-tuning. To solve the above problems, we propose a learning from demonstrations method named A-SILfD, which treats expert demonstrations as the agent's successful experiences and uses experiences to constrain policy improvement. Furthermore, we prevent performance degradation due to large estimation errors in the Q-function by the ensemble Q-functions. Our experiments show that A-SILfD can significantly improve sample efficiency using a small number of different quality expert demonstrations. In four Mujoco continuous control tasks, A-SILfD can significantly outperform baseline methods after 150,000 steps of online training and is not misled by imperfect expert demonstrations during training.