Maximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding Errors
This work addresses the problem of compounding errors in model-based reinforcement learning for researchers and practitioners, offering an incremental improvement over existing methods.
The paper tackles the challenge of compounding errors in model-based reinforcement learning by proposing Maximum Entropy Model Rollouts (MEMR), which uses single-step rollouts and maximizes diversity through non-uniform state sampling. The approach achieves sample efficiency comparable to model-based algorithms, matches asymptotic performance of model-free methods, and reduces computational requirements in locomotion benchmarks.
Model usage is the central challenge of model-based reinforcement learning. Although dynamics model based on deep neural networks provide good generalization for single step prediction, such ability is over exploited when it is used to predict long horizon trajectories due to compounding errors. In this work, we propose a Dyna-style model-based reinforcement learning algorithm, which we called Maximum Entropy Model Rollouts (MEMR). To eliminate the compounding errors, we only use our model to generate single-step rollouts. Furthermore, we propose to generate \emph{diverse} model rollouts by non-uniform sampling of the environment states such that the entropy of the model rollouts is maximized. We mathematically derived the maximum entropy sampling criteria for one data case under Gaussian prior. To accomplish this criteria, we propose to utilize a prioritized experience replay. Our preliminary experiments in challenging locomotion benchmarks show that our approach achieves the same sample efficiency of the best model-based algorithms, matches the asymptotic performance of the best model-free algorithms, and significantly reduces the computation requirements of other model-based methods.