Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contexts
This addresses generalization issues in reinforcement learning for real-world applications where diverse training data is scarce, though it appears incremental as it builds on existing CMDP frameworks.
The paper tackles the problem of poor generalization in deep reinforcement learning to new environments by proposing a context-enhanced Bellman equation and a data augmentation method, showing improved generalization with numerical validation in simulations.
Deep reinforcement learning (DRL) has achieved remarkable success across multiple domains, including competitive games, natural language processing, and robotics. Despite these advancements, policies trained via DRL often struggle to generalize to evaluation environments with different parameters. This challenge is typically addressed by training with multiple contexts and/or by leveraging additional structure in the problem. However, obtaining sufficient training data across diverse contexts can be impractical in real-world applications. In this work, we consider contextual Markov decision processes (CMDPs) with transition and reward functions that exhibit regularity in context parameters. We introduce the context-enhanced Bellman equation (CEBE) to improve generalization when training on a single context. We prove both analytically and empirically that the CEBE yields a first-order approximation to the Q-function trained across multiple contexts. We then derive context sample enhancement (CSE) as an efficient data augmentation method for approximating the CEBE in deterministic control environments. We numerically validate the performance of CSE in simulation environments, showcasing its potential to improve generalization in DRL.