On-Policy Model Errors in Reinforcement Learning
This addresses data inefficiency and instability in reinforcement learning for robotics and simulation tasks, representing an incremental improvement over existing model-based methods.
The paper tackles the problem of model errors and data inefficiency in reinforcement learning by combining real-world data with a learned model to improve policy gradients, showing that the method drastically improves existing model-based approaches on MuJoCo- and PyBullet-benchmarks.
Model-free reinforcement learning algorithms can compute policy gradients given sampled environment transitions, but require large amounts of data. In contrast, model-based methods can use the learned model to generate new data, but model errors and bias can render learning unstable or suboptimal. In this paper, we present a novel method that combines real-world data and a learned model in order to get the best of both worlds. The core idea is to exploit the real-world data for on-policy predictions and use the learned model only to generalize to different actions. Specifically, we use the data as time-dependent on-policy correction terms on top of a learned model, to retain the ability to generate data without accumulating errors over long prediction horizons. We motivate this method theoretically and show that it counteracts an error term for model-based policy improvement. Experiments on MuJoCo- and PyBullet-benchmarks show that our method can drastically improve existing model-based approaches without introducing additional tuning parameters.