Behavior Contrastive Learning for Unsupervised Skill Discovery
This addresses the issue of simple and static skills in unsupervised skill discovery for reinforcement learning, offering an incremental improvement over previous mutual information-based methods.
The paper tackles the problem of learning diverse skills in reinforcement learning without extrinsic rewards by proposing a behavior contrastive learning method, which results in generating diverse and far-reaching skills and achieving competitive performance in downstream tasks compared to state-of-the-art methods.
In reinforcement learning, unsupervised skill discovery aims to learn diverse skills without extrinsic rewards. Previous methods discover skills by maximizing the mutual information (MI) between states and skills. However, such an MI objective tends to learn simple and static skills and may hinder exploration. In this paper, we propose a novel unsupervised skill discovery method through contrastive learning among behaviors, which makes the agent produce similar behaviors for the same skill and diverse behaviors for different skills. Under mild assumptions, our objective maximizes the MI between different behaviors based on the same skill, which serves as an upper bound of the previous MI objective. Meanwhile, our method implicitly increases the state entropy to obtain better state coverage. We evaluate our method on challenging mazes and continuous control tasks. The results show that our method generates diverse and far-reaching skills, and also obtains competitive performance in downstream tasks compared to the state-of-the-art methods.