On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learning
This work addresses the lack of theoretical convergence analysis for meta-RL methods, which is important for researchers in reinforcement learning and meta-learning, though it is incremental as it builds on existing MAML frameworks.
The paper tackles the problem of providing convergence guarantees for model-agnostic meta-reinforcement learning algorithms by proposing SG-MRL, a variant of MAML, and derives iteration and sample complexity bounds for finding an ε-first-order stationary point, with empirical comparisons in deep RL environments.
We consider Model-Agnostic Meta-Learning (MAML) methods for Reinforcement Learning (RL) problems, where the goal is to find a policy using data from several tasks represented by Markov Decision Processes (MDPs) that can be updated by one step of stochastic policy gradient for the realized MDP. In particular, using stochastic gradients in MAML update steps is crucial for RL problems since computation of exact gradients requires access to a large number of possible trajectories. For this formulation, we propose a variant of the MAML method, named Stochastic Gradient Meta-Reinforcement Learning (SG-MRL), and study its convergence properties. We derive the iteration and sample complexity of SG-MRL to find an $ε$-first-order stationary point, which, to the best of our knowledge, provides the first convergence guarantee for model-agnostic meta-reinforcement learning algorithms. We further show how our results extend to the case where more than one step of stochastic policy gradient method is used at test time. Finally, we empirically compare SG-MRL and MAML in several deep RL environments.