Reusable Options through Gradient-based Meta Learning
This work addresses the challenge of fast adaptation in reinforcement learning for agents, though it appears incremental as it builds on existing deep learning methods for options.
The paper tackled the problem of learning reusable temporal abstractions (options) in hierarchical reinforcement learning to accelerate new task learning, and proposed a gradient-based meta-learning approach that outperformed prior methods in learning transferable components.
Hierarchical methods in reinforcement learning have the potential to reduce the amount of decisions that the agent needs to perform when learning new tasks. However, finding reusable useful temporal abstractions that facilitate fast learning remains a challenging problem. Recently, several deep learning approaches were proposed to learn such temporal abstractions in the form of options in an end-to-end manner. In this work, we point out several shortcomings of these methods and discuss their potential negative consequences. Subsequently, we formulate the desiderata for reusable options and use these to frame the problem of learning options as a gradient-based meta-learning problem. This allows us to formulate an objective that explicitly incentivizes options which allow a higher-level decision maker to adjust in few steps to different tasks. Experimentally, we show that our method is able to learn transferable components which accelerate learning and performs better than existing prior methods developed for this setting. Additionally, we perform ablations to quantify the impact of using gradient-based meta-learning as well as other proposed changes.