Continual Learning In Environments With Polynomial Mixing Times
This addresses a fundamental bottleneck for continual RL practitioners dealing with complex environments, though it appears incremental as it builds on existing mixing time theory.
The paper tackles the problem of how polynomial mixing times in Markov decision processes limit performance in continual reinforcement learning, demonstrating both theoretically and empirically that these mixing times emerge in practice and lead to unstable learning behaviors like catastrophic forgetting.
The mixing time of the Markov chain induced by a policy limits performance in real-world continual learning scenarios. Yet, the effect of mixing times on learning in continual reinforcement learning (RL) remains underexplored. In this paper, we characterize problems that are of long-term interest to the development of continual RL, which we call scalable MDPs, through the lens of mixing times. In particular, we theoretically establish that scalable MDPs have mixing times that scale polynomially with the size of the problem. We go on to demonstrate that polynomial mixing times present significant difficulties for existing approaches, which suffer from myopic bias and stale bootstrapped estimates. To validate our theory, we study the empirical scaling behavior of mixing times with respect to the number of tasks and task duration for high performing policies deployed across multiple Atari games. Our analysis demonstrates both that polynomial mixing times do emerge in practice and how their existence may lead to unstable learning behavior like catastrophic forgetting in continual learning settings.