Dyna Planning using a Feature Based Generative Model
This work addresses a bottleneck in reinforcement learning for environments with limited real data, offering a domain-specific improvement.
The paper tackled the problem of learning a good generative model for Dyna-style reinforcement learning in large state spaces, such as those with image observations, and demonstrated that using deep belief networks significantly outperforms linear expectation models.
Dyna-style reinforcement learning is a powerful approach for problems where not much real data is available. The main idea is to supplement real trajectories, or sequences of sampled states over time, with simulated ones sampled from a learned model of the environment. However, in large state spaces, the problem of learning a good generative model of the environment has been open so far. We propose to use deep belief networks to learn an environment model for use in Dyna. We present our approach and validate it empirically on problems where the state observations consist of images. Our results demonstrate that using deep belief networks, which are full generative models, significantly outperforms the use of linear expectation models, proposed in Sutton et al. (2008)