Demonstration-efficient Inverse Reinforcement Learning in Procedurally Generated Environments
This work provides a demonstration-efficient method for applying Inverse Reinforcement Learning in procedurally generated environments, which is beneficial for researchers and developers working on generalization and domain shift in AI agents.
This paper addresses the challenge of applying Inverse Reinforcement Learning (IRL) to procedurally generated environments, which typically require a large number of expert demonstrations. The authors propose DE-AIRL, a demonstration-efficient technique based on Adversarial Inverse Reinforcement Learning, which significantly reduces the need for demonstrations by using a limited set of initial seed levels and training modifications. This method successfully extrapolates reward functions that generalize to fully procedural domains, demonstrated on MiniGrid and DeepCrawl.
Deep Reinforcement Learning achieves very good results in domains where reward functions can be manually engineered. At the same time, there is growing interest within the community in using games based on Procedurally Content Generation (PCG) as benchmark environments since this type of environment is perfect for studying overfitting and generalization of agents under domain shift. Inverse Reinforcement Learning (IRL) can instead extrapolate reward functions from expert demonstrations, with good results even on high-dimensional problems, however there are no examples of applying these techniques to procedurally-generated environments. This is mostly due to the number of demonstrations needed to find a good reward model. We propose a technique based on Adversarial Inverse Reinforcement Learning which can significantly decrease the need for expert demonstrations in PCG games. Through the use of an environment with a limited set of initial seed levels, plus some modifications to stabilize training, we show that our approach, DE-AIRL, is demonstration-efficient and still able to extrapolate reward functions which generalize to the fully procedural domain. We demonstrate the effectiveness of our technique on two procedural environments, MiniGrid and DeepCrawl, for a variety of tasks.