Learning and Querying Fast Generative Models for Reinforcement Learning
This addresses the problem of high computational costs in model-based RL for researchers and practitioners, though it is incremental as it builds on existing generative model approaches.
The paper tackled the challenge of synthesizing computationally efficient and accurate environment models in model-based reinforcement learning by using state-space models that learn on compact state representations, resulting in agents that outperform strong model-free baselines on the game MSPACMAN.
A key challenge in model-based reinforcement learning (RL) is to synthesize computationally efficient and accurate environment models. We show that carefully designed generative models that learn and operate on compact state representations, so-called state-space models, substantially reduce the computational costs for predicting outcomes of sequences of actions. Extensive experiments establish that state-space models accurately capture the dynamics of Atari games from the Arcade Learning Environment from raw pixels. The computational speed-up of state-space models while maintaining high accuracy makes their application in RL feasible: We demonstrate that agents which query these models for decision making outperform strong model-free baselines on the game MSPACMAN, demonstrating the potential of using learned environment models for planning.