Adaptive Rollout Length for Model-Based RL Using Model-Free Deep RL
This addresses the efficiency and performance trade-off in model-based RL for researchers and practitioners, though it is incremental as it adapts an existing hyperparameter tuning approach.
The paper tackles the problem of tuning the rollout length hyperparameter in model-based reinforcement learning, which balances prediction quality and efficiency, by framing it as a meta-level decision problem solved with model-free deep RL, resulting in outperformance over heuristic baselines on two RL environments.
Model-based reinforcement learning promises to learn an optimal policy from fewer interactions with the environment compared to model-free reinforcement learning by learning an intermediate model of the environment in order to predict future interactions. When predicting a sequence of interactions, the rollout length, which limits the prediction horizon, is a critical hyperparameter as accuracy of the predictions diminishes in the regions that are further away from real experience. As a result, with a longer rollout length, an overall worse policy is learned in the long run. Thus, the hyperparameter provides a trade-off between quality and efficiency. In this work, we frame the problem of tuning the rollout length as a meta-level sequential decision-making problem that optimizes the final policy learned by model-based reinforcement learning given a fixed budget of environment interactions by adapting the hyperparameter dynamically based on feedback from the learning process, such as accuracy of the model and the remaining budget of interactions. We use model-free deep reinforcement learning to solve the meta-level decision problem and demonstrate that our approach outperforms common heuristic baselines on two well-known reinforcement learning environments.