Learning Dynamics Models for Model Predictive Agents
This work addresses the challenge of evaluating and optimizing design choices in model-based reinforcement learning for researchers and practitioners, though it is incremental as it builds on existing methods without introducing new paradigms.
The paper investigates the impact of various design choices in learning dynamics models for model-based reinforcement learning, comparing their performance to planning with a ground-truth simulator across 5 domains of the DeepMind Control Suite, and provides qualitative findings and rules of thumb for improving model-based planning.
Model-Based Reinforcement Learning involves learning a \textit{dynamics model} from data, and then using this model to optimise behaviour, most often with an online \textit{planner}. Much of the recent research along these lines presents a particular set of design choices, involving problem definition, model learning and planning. Given the multiple contributions, it is difficult to evaluate the effects of each. This paper sets out to disambiguate the role of different design choices for learning dynamics models, by comparing their performance to planning with a ground-truth model -- the simulator. First, we collect a rich dataset from the training sequence of a model-free agent on 5 domains of the DeepMind Control Suite. Second, we train feed-forward dynamics models in a supervised fashion, and evaluate planner performance while varying and analysing different model design choices, including ensembling, stochasticity, multi-step training and timestep size. Besides the quantitative analysis, we describe a set of qualitative findings, rules of thumb, and future research directions for planning with learned dynamics models. Videos of the results are available at https://sites.google.com/view/learning-better-models.