Exploring Beyond-Demonstrator via Meta Learning-Based Reward Extrapolation
This work addresses the challenge of limited training data in beyond-demonstrator imitation learning, which is a problem for researchers and practitioners aiming to outperform demonstrators with scarce resources.
This paper tackles the problem of extrapolating beyond-demonstrator performance in imitation learning with limited demonstrations. The authors propose a meta learning-based reward extrapolation (MLRE) algorithm that learns an initial reward function from data-rich tasks and fine-tunes it for the target task, achieving impressive performance improvement over similar algorithms.
Extrapolating beyond-demonstrator (BD) performance through the imitation learning (IL) algorithm aims to learn from and subsequently outperform the demonstrator. To that end, a representative approach is to leverage inverse reinforcement learning (IRL) to infer a reward function from demonstrations before performing RL on the learned reward function. However, most existing reward extrapolation methods require massive demonstrations, making it difficult to be applied in tasks of limited training data. To address this problem, one simple solution is to perform data augmentation to artificially generate more training data, which may incur severe inductive bias and policy performance loss. In this paper, we propose a novel meta learning-based reward extrapolation (MLRE) algorithm, which can effectively approximate the ground-truth rewards using limited demonstrations. More specifically, MLRE first learns an initial reward function from a set of tasks that have abundant training data. Then the learned reward function will be fine-tuned using data of the target task. Extensive simulation results demonstrated that the proposed MLRE can achieve impressive performance improvement as compared to other similar BDIL algorithms.