Continual Task Learning through Adaptive Policy Self-Composition
This work addresses the problem of catastrophic forgetting in continual offline RL for long-lived agents, offering a novel method that is incremental but shows strong gains in specific scenarios.
The paper tackles the challenge of continual offline reinforcement learning (CORL) by introducing the Offline Continual World benchmark and showing that traditional continual learning methods fail due to distribution shifts. They propose CompoFormer, a transformer model that adaptively composes previous policies, which outperforms conventional methods, especially in longer task sequences, achieving a balance between plasticity and stability.
Training a generalizable agent to continually learn a sequence of tasks from offline trajectories is a natural requirement for long-lived agents, yet remains a significant challenge for current offline reinforcement learning (RL) algorithms. Specifically, an agent must be able to rapidly adapt to new tasks using newly collected trajectories (plasticity), while retaining knowledge from previously learned tasks (stability). However, systematic analyses of this setting are scarce, and it remains unclear whether conventional continual learning (CL) methods are effective in continual offline RL (CORL) scenarios. In this study, we develop the Offline Continual World benchmark and demonstrate that traditional CL methods struggle with catastrophic forgetting, primarily due to the unique distribution shifts inherent to CORL scenarios. To address this challenge, we introduce CompoFormer, a structure-based continual transformer model that adaptively composes previous policies via a meta-policy network. Upon encountering a new task, CompoFormer leverages semantic correlations to selectively integrate relevant prior policies alongside newly trained parameters, thereby enhancing knowledge sharing and accelerating the learning process. Our experiments reveal that CompoFormer outperforms conventional CL methods, particularly in longer task sequences, showcasing a promising balance between plasticity and stability.