Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning
This addresses catastrophic forgetting in continual RL, an incremental improvement for AI systems requiring adaptation to new tasks.
The study tackled continual reinforcement learning by proposing a dual-learner framework with a fast learner for knowledge transfer and a meta learner for integration, showing superior performance in pixel-based and continuous control benchmarks compared to baselines.
Inspired by the human learning and memory system, particularly the interplay between the hippocampus and cerebral cortex, this study proposes a dual-learner framework comprising a fast learner and a meta learner to address continual Reinforcement Learning~(RL) problems. These two learners are coupled to perform distinct yet complementary roles: the fast learner focuses on knowledge transfer, while the meta learner ensures knowledge integration. In contrast to traditional multi-task RL approaches that share knowledge through average return maximization, our meta learner incrementally integrates new experiences by explicitly minimizing catastrophic forgetting, thereby supporting efficient cumulative knowledge transfer for the fast learner. To facilitate rapid adaptation in new environments, we introduce an adaptive meta warm-up mechanism that selectively harnesses past knowledge. We conduct experiments in various pixel-based and continuous control benchmarks, revealing the superior performance of continual learning for our proposed dual-learner approach relative to baseline methods. The code is released in https://github.com/datake/FAME.