The effects of negative adaptation in Model-Agnostic Meta-Learning
This highlights a critical flaw in meta-learning for reinforcement learning, potentially affecting researchers and practitioners relying on these methods for few-shot learning.
The paper demonstrates that in meta-reinforcement learning, adaptation in algorithms like MAML can significantly decrease performance on tasks, even those seen during meta-training, challenging the assumption that adaptation always improves results.
The capacity of meta-learning algorithms to quickly adapt to a variety of tasks, including ones they did not experience during meta-training, has been a key factor in the recent success of these methods on few-shot learning problems. This particular advantage of using meta-learning over standard supervised or reinforcement learning is only well founded under the assumption that the adaptation phase does improve the performance of our model on the task of interest. However, in the classical framework of meta-learning, this constraint is only mildly enforced, if not at all, and we only see an improvement on average over a distribution of tasks. In this paper, we show that the adaptation in an algorithm like MAML can significantly decrease the performance of an agent in a meta-reinforcement learning setting, even on a range of meta-training tasks.