Provable Zero-Shot Generalization in Offline Reinforcement Learning
This work addresses the challenge of generalization in offline RL for AI systems that need to adapt to new environments without additional data, representing a foundational step in understanding this phenomenon.
The paper tackles the problem of offline reinforcement learning with zero-shot generalization, where an agent must learn from offline data across multiple training environments to perform well on unseen test environments without further interaction. The authors propose PERM and PPPO methods, which use pessimistic policy evaluation to achieve near-optimal policies with zero-shot generalization.
In this work, we study offline reinforcement learning (RL) with zero-shot generalization property (ZSG), where the agent has access to an offline dataset including experiences from different environments, and the goal of the agent is to train a policy over the training environments which performs well on test environments without further interaction. Existing work showed that classical offline RL fails to generalize to new, unseen environments. We propose pessimistic empirical risk minimization (PERM) and pessimistic proximal policy optimization (PPPO), which leverage pessimistic policy evaluation to guide policy learning and enhance generalization. We show that both PERM and PPPO are capable of finding a near-optimal policy with ZSG. Our result serves as a first step in understanding the foundation of the generalization phenomenon in offline reinforcement learning.