Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
This addresses a key bottleneck in data-efficient reinforcement learning for applications requiring safe or penalized actions, though it is an incremental improvement over prior optimistic methods.
The paper tackles the problem of insufficient exploration in model-based reinforcement learning by proposing H-UCRL, a practical optimistic exploration algorithm that hallucinates control on epistemic uncertainty, enabling use with standard planners. Experiments show it significantly speeds up learning in settings with action penalties, where existing methods struggle.
Model-based reinforcement learning algorithms with probabilistic dynamical models are amongst the most data-efficient learning methods. This is often attributed to their ability to distinguish between epistemic and aleatoric uncertainty. However, while most algorithms distinguish these two uncertainties for learning the model, they ignore it when optimizing the policy, which leads to greedy and insufficient exploration. At the same time, there are no practical solvers for optimistic exploration algorithms. In this paper, we propose a practical optimistic exploration algorithm (H-UCRL). H-UCRL reparameterizes the set of plausible models and hallucinates control directly on the epistemic uncertainty. By augmenting the input space with the hallucinated inputs, H-UCRL can be solved using standard greedy planners. Furthermore, we analyze H-UCRL and construct a general regret bound for well-calibrated models, which is provably sublinear in the case of Gaussian Process models. Based on this theoretical foundation, we show how optimistic exploration can be easily combined with state-of-the-art reinforcement learning algorithms and different probabilistic models. Our experiments demonstrate that optimistic exploration significantly speeds-up learning when there are penalties on actions, a setting that is notoriously difficult for existing model-based reinforcement learning algorithms.