Hierarchical Subspaces of Policies for Continual Offline Reinforcement Learning
This addresses forgetting and scalability issues in continual learning for autonomous robotics and video game navigation, though it appears incremental as it builds on existing hierarchical and subspace methods.
The paper tackles the problem of continual reinforcement learning where agents must adapt to new tasks while retaining previous skills, introducing HiSPO, a hierarchical framework using policy subspaces for navigation tasks from offline data. Results show competitive performance in MuJoCo maze and video game simulations with improved memory usage and efficiency.
We consider a Continual Reinforcement Learning setup, where a learning agent must continuously adapt to new tasks while retaining previously acquired skill sets, with a focus on the challenge of avoiding forgetting past gathered knowledge and ensuring scalability with the growing number of tasks. Such issues prevail in autonomous robotics and video game simulations, notably for navigation tasks prone to topological or kinematic changes. To address these issues, we introduce HiSPO, a novel hierarchical framework designed specifically for continual learning in navigation settings from offline data. Our method leverages distinct policy subspaces of neural networks to enable flexible and efficient adaptation to new tasks while preserving existing knowledge. We demonstrate, through a careful experimental study, the effectiveness of our method in both classical MuJoCo maze environments and complex video game-like navigation simulations, showcasing competitive performances and satisfying adaptability with respect to classical continual learning metrics, in particular regarding the memory usage and efficiency.