Calibrated Model-Based Deep Reinforcement Learning
This work addresses a key bottleneck in model-based reinforcement learning for AI systems, offering a simple and efficient solution to improve performance with minimal overhead, though it is incremental in nature.
The paper tackles the problem of inaccurate predictive uncertainties in model-based deep reinforcement learning by proposing calibrated models, which match predicted probabilities with empirical frequencies, and demonstrates that this approach improves planning, sample complexity, and exploration, achieving state-of-the-art performance on the HalfCheetah MuJoCo task with 50% fewer samples than the leading method.
Estimates of predictive uncertainty are important for accurate model-based planning and reinforcement learning. However, predictive uncertainties---especially ones derived from modern deep learning systems---can be inaccurate and impose a bottleneck on performance. This paper explores which uncertainties are needed for model-based reinforcement learning and argues that good uncertainties must be calibrated, i.e. their probabilities should match empirical frequencies of predicted events. We describe a simple way to augment any model-based reinforcement learning agent with a calibrated model and show that doing so consistently improves planning, sample complexity, and exploration. On the \textsc{HalfCheetah} MuJoCo task, our system achieves state-of-the-art performance using 50\% fewer samples than the current leading approach. Our findings suggest that calibration can improve the performance of model-based reinforcement learning with minimal computational and implementation overhead.