A Model-based Approach for Sample-efficient Multi-task Reinforcement Learning
This work addresses the challenge of quick adaptation to new tasks with limited samples in reinforcement learning, offering a model-based approach that is incremental over existing meta-learning methods.
The paper tackles the problem of sample-efficient adaptation in multi-task reinforcement learning by learning a dynamical model during training and using it for policy optimization in a virtual environment, requiring significantly fewer samples. It demonstrates efficacy over MAML on continuous control benchmarks, showing improved performance with reduced sample usage.
The aim of multi-task reinforcement learning is two-fold: (1) efficiently learn by training against multiple tasks and (2) quickly adapt, using limited samples, to a variety of new tasks. In this work, the tasks correspond to reward functions for environments with the same (or similar) dynamical models. We propose to learn a dynamical model during the training process and use this model to perform sample-efficient adaptation to new tasks at test time. We use significantly fewer samples by performing policy optimization only in a "virtual" environment whose transitions are given by our learned dynamical model. Our algorithm sequentially trains against several tasks. Upon encountering a new task, we first warm-up a policy on our learned dynamical model, which requires no new samples from the environment. We then adapt the dynamical model with samples from this policy in the real environment. We evaluate our approach on several continuous control benchmarks and demonstrate its efficacy over MAML, a state-of-the-art meta-learning algorithm, on these tasks.