Learning Diverse Options via InfoMax Termination Critic
This work addresses the challenge of learning reusable temporally extended actions for unknown task distributions in reinforcement learning, representing an incremental improvement over existing mutual information-based skill learning methods.
The paper tackles the problem of learning reusable options in reinforcement learning by proposing the InfoMax Termination Critic (IMTC) algorithm, which maximizes mutual information between options and state transitions to improve option diversity, and demonstrates that IMTC significantly enhances diversity and aids quick adaptation in complex tasks like object manipulation.
We consider the problem of autonomously learning reusable temporally extended actions, or options, in reinforcement learning. While options can speed up transfer learning by serving as reusable building blocks, learning reusable options for unknown task distribution remains challenging. Motivated by the recent success of mutual information (MI) based skill learning, we hypothesize that more diverse options are more reusable. To this end, we propose a method for learning termination conditions of options by maximizing MI between options and corresponding state transitions. We derive a scalable approximation of this MI maximization via gradient ascent, yielding the InfoMax Termination Critic (IMTC) algorithm. Our experiments demonstrate that IMTC significantly improves the diversity of learned options without extrinsic rewards combined with an intrinsic option learning method. Moreover, we test the reusability of learned options by transferring options into various tasks, confirming that IMTC helps quick adaptation, especially in complex domains where an agent needs to manipulate objects.