Model Primitive Hierarchical Lifelong Reinforcement Learning
This addresses the difficulty of task decomposition in hierarchical reinforcement learning without requiring predefined task distributions, though it appears incremental as it builds on existing hierarchical and meta-learning approaches.
The paper tackles the problem of learning interpretable and transferable subpolicies from a single complex task by proposing a framework that uses diverse suboptimal world models to automatically decompose tasks into modular subpolicies with a coordinating controller, demonstrating effectiveness on high-dimensional continuous control tasks for both single-task and lifelong learning.
Learning interpretable and transferable subpolicies and performing task decomposition from a single, complex task is difficult. Some traditional hierarchical reinforcement learning techniques enforce this decomposition in a top-down manner, while meta-learning techniques require a task distribution at hand to learn such decompositions. This paper presents a framework for using diverse suboptimal world models to decompose complex task solutions into simpler modular subpolicies. This framework performs automatic decomposition of a single source task in a bottom up manner, concurrently learning the required modular subpolicies as well as a controller to coordinate them. We perform a series of experiments on high dimensional continuous action control tasks to demonstrate the effectiveness of this approach at both complex single task learning and lifelong learning. Finally, we perform ablation studies to understand the importance and robustness of different elements in the framework and limitations to this approach.