Intrinsic Reward Driven Imitation Learning via Generative Model
This addresses the problem of improving imitation learning for AI agents in complex environments like Atari, offering a method that can outperform human demonstrators, which is an incremental advance over existing IRL approaches.
The paper tackles the challenge of imitation learning in high-dimensional environments where most inverse reinforcement learning methods fail to outperform the demonstrator, proposing a novel reward learning module using a generative model that achieves up to 5 times the performance of the demonstration on multiple Atari games.
Imitation learning in a high-dimensional environment is challenging. Most inverse reinforcement learning (IRL) methods fail to outperform the demonstrator in such a high-dimensional environment, e.g., Atari domain. To address this challenge, we propose a novel reward learning module to generate intrinsic reward signals via a generative model. Our generative method can perform better forward state transition and backward action encoding, which improves the module's dynamics modeling ability in the environment. Thus, our module provides the imitation agent both the intrinsic intention of the demonstrator and a better exploration ability, which is critical for the agent to outperform the demonstrator. Empirical results show that our method outperforms state-of-the-art IRL methods on multiple Atari games, even with one-life demonstration. Remarkably, our method achieves performance that is up to 5 times the performance of the demonstration.