Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn
This addresses the issue of adaptability in continual RL for AI agents, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackled the problem of plasticity loss in continual reinforcement learning by analyzing churn, showing that reducing churn prevents rank collapse and improves learning performance, with C-CHAIN outperforming baselines across multiple benchmarks.
Plasticity, or the ability of an agent to adapt to new tasks, environments, or distributions, is crucial for continual learning. In this paper, we study the loss of plasticity in deep continual RL from the lens of churn: network output variability for out-of-batch data induced by mini-batch training. We demonstrate that (1) the loss of plasticity is accompanied by the exacerbation of churn due to the gradual rank decrease of the Neural Tangent Kernel (NTK) matrix; (2) reducing churn helps prevent rank collapse and adjusts the step size of regular RL gradients adaptively. Moreover, we introduce Continual Churn Approximated Reduction (C-CHAIN) and demonstrate it improves learning performance and outperforms baselines in a diverse range of continual learning environments on OpenAI Gym Control, ProcGen, DeepMind Control Suite, and MinAtar benchmarks.