Evolutionary Planning in Latent Space
This work addresses the challenge of applying planning to real-world reinforcement learning problems where explicit world models are absent, offering an incremental improvement for researchers and practitioners in RL.
This paper tackles the problem of planning in reinforcement learning when a world model is not available. They propose learning a world model using a VAE and MDRNN, and then using evolutionary planning (RMHC) in this learned latent space. The resulting planning agents outperform standard model-free reinforcement learning approaches.
Planning is a powerful approach to reinforcement learning with several desirable properties. However, it requires a model of the world, which is not readily available in many real-life problems. In this paper, we propose to learn a world model that enables Evolutionary Planning in Latent Space (EPLS). We use a Variational Auto Encoder (VAE) to learn a compressed latent representation of individual observations and extend a Mixture Density Recurrent Neural Network (MDRNN) to learn a stochastic, multi-modal forward model of the world that can be used for planning. We use the Random Mutation Hill Climbing (RMHC) to find a sequence of actions that maximize expected reward in this learned model of the world. We demonstrate how to build a model of the world by bootstrapping it with rollouts from a random policy and iteratively refining it with rollouts from an increasingly accurate planning policy using the learned world model. After a few iterations of this refinement, our planning agents are better than standard model-free reinforcement learning approaches demonstrating the viability of our approach.