Block Contextual MDPs for Continual Learning
This work addresses the challenge of continual learning in RL for scenarios where environment dynamics change over time, though it appears incremental as it builds on existing MDP frameworks.
The authors tackled the problem of nonstationarity in reinforcement learning by proposing the block contextual MDP framework, which relaxes the stationarity assumption and enables zero-shot adaptation, achieving strong performance on nonstationary environments.
In reinforcement learning (RL), when defining a Markov Decision Process (MDP), the environment dynamics is implicitly assumed to be stationary. This assumption of stationarity, while simplifying, can be unrealistic in many scenarios. In the continual reinforcement learning scenario, the sequence of tasks is another source of nonstationarity. In this work, we propose to examine this continual reinforcement learning setting through the block contextual MDP (BC-MDP) framework, which enables us to relax the assumption of stationarity. This framework challenges RL algorithms to handle both nonstationarity and rich observation settings and, by additionally leveraging smoothness properties, enables us to study generalization bounds for this setting. Finally, we take inspiration from adaptive control to propose a novel algorithm that addresses the challenges introduced by this more realistic BC-MDP setting, allows for zero-shot adaptation at evaluation time, and achieves strong performance on several nonstationary environments.