Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal
This addresses the problem of adapting to changing tasks in real-world applications like robotic control for reinforcement learning practitioners, though it is incremental as it builds on existing diffusion models with rehearsal techniques.
The paper tackles the challenge of plasticity-stability trade-off in continual offline reinforcement learning by proposing Continual Diffuser (CoD), which uses experience rehearsal to adapt to new tasks while retaining knowledge, achieving promising performance and outperforming existing methods on most tasks.
Artificial neural networks, especially recent diffusion-based models, have shown remarkable superiority in gaming, control, and QA systems, where the training tasks' datasets are usually static. However, in real-world applications, such as robotic control of reinforcement learning (RL), the tasks are changing, and new tasks arise in a sequential order. This situation poses the new challenge of plasticity-stability trade-off for training an agent who can adapt to task changes and retain acquired knowledge. In view of this, we propose a rehearsal-based continual diffusion model, called Continual Diffuser (CoD), to endow the diffuser with the capabilities of quick adaptation (plasticity) and lasting retention (stability). Specifically, we first construct an offline benchmark that contains 90 tasks from multiple domains. Then, we train the CoD on each task with sequential modeling and conditional generation for making decisions. Next, we preserve a small portion of previous datasets as the rehearsal buffer and replay it to retain the acquired knowledge. Extensive experiments on a series of tasks show CoD can achieve a promising plasticity-stability trade-off and outperform existing diffusion-based methods and other representative baselines on most tasks.