Shaping Sparse Rewards in Reinforcement Learning: A Semi-supervised Approach
This addresses the problem of sparse rewards for reinforcement learning agents in domains like Atari and robotics, offering an incremental improvement over existing supervised methods.
The paper tackles the challenge of sparse rewards in reinforcement learning by introducing a semi-supervised approach that uses data augmentation to learn from zero-reward transitions, resulting in up to twice the peak scores in sparse environments and a 15.8% improvement over other methods.
In many real-world scenarios, reward signal for agents are exceedingly sparse, making it challenging to learn an effective reward function for reward shaping. To address this issue, the proposed approach in this paper performs reward shaping not only by utilizing non-zero-reward transitions but also by employing the \emph{Semi-Supervised Learning} (SSL) technique combined with a novel data augmentation to learn trajectory space representations from the majority of transitions, {i.e}., zero-reward transitions, thereby improving the efficacy of reward shaping. Experimental results in Atari and robotic manipulation demonstrate that our method outperforms supervised-based approaches in reward inference, leading to higher agent scores. Notably, in more sparse-reward environments, our method achieves up to twice the peak scores compared to supervised baselines. The proposed double entropy data augmentation enhances performance, showcasing a 15.8\% increase in best score over other augmentation methods