Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards
This addresses a practical limitation in robotics for researchers and engineers by providing a method to handle sparse rewards, though it appears incremental as it builds on self-imitation learning.
The paper tackles the problem of applying reinforcement learning in robotic control with sparse and delayed rewards by proposing Self-Imitation Learning with Constant Reward (SILCR), which assigns constant immediate rewards based on final episodic rewards, and results show it significantly outperforms alternatives in such tasks and achieves competitive performance even against methods with dense rewards.
The application of reinforcement learning (RL) in robotic control is still limited in the environments with sparse and delayed rewards. In this paper, we propose a practical self-imitation learning method named Self-Imitation Learning with Constant Reward (SILCR). Instead of requiring hand-defined immediate rewards from environments, our method assigns the immediate rewards at each timestep with constant values according to their final episodic rewards. In this way, even if the dense rewards from environments are unavailable, every action taken by the agents would be guided properly. We demonstrate the effectiveness of our method in some challenging continuous robotics control tasks in MuJoCo simulation and the results show that our method significantly outperforms the alternative methods in tasks with sparse and delayed rewards. Even compared with alternatives with dense rewards available, our method achieves competitive performance. The ablation experiments also show the stability and reproducibility of our method.