Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator
This work addresses the challenge of data efficiency and generalizability in reinforcement learning for researchers and practitioners, though it appears incremental as it builds on existing meta-RL analyses.
The paper tackles the problem of meta-reinforcement learning by developing a bilevel optimization framework (BO-MRL) to learn a meta-prior for task-specific policy adaptation, providing provable upper bounds on the optimality gap and demonstrating superior effectiveness over benchmarks.
Meta-reinforcement learning (Meta-RL) has attracted attention due to its capability to enhance reinforcement learning (RL) algorithms, in terms of data efficiency and generalizability. In this paper, we develop a bilevel optimization framework for meta-RL (BO-MRL) to learn the meta-prior for task-specific policy adaptation, which implements multiple-step policy optimization on one-time data collection. Beyond existing meta-RL analyses, we provide upper bounds of the expected optimality gap over the task distribution. This metric measures the distance of the policy adaptation from the learned meta-prior to the task-specific optimum, and quantifies the model's generalizability to the task distribution. We empirically validate the correctness of the derived upper bounds and demonstrate the superior effectiveness of the proposed algorithm over benchmarks.