Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation
This work addresses a bottleneck in reinforcement learning for robotics by enhancing sample efficiency, though it is incremental as it builds on existing offline pre-training methods.
The paper tackles the problem of limited and low-quality offline datasets degrading online reinforcement learning performance by proposing a model-based data augmentation strategy for offline pre-training. The results show that this approach can jump-start online fine-tuning and reduce required environment interactions by an order of magnitude in some cases.
Offline reinforcement learning leverages pre-collected datasets of transitions to train policies. It can serve as effective initialization for online algorithms, enhancing sample efficiency and speeding up convergence. However, when such datasets are limited in size and quality, offline pre-training can produce sub-optimal policies and lead to degraded online reinforcement learning performance. In this paper we propose a model-based data augmentation strategy to maximize the benefits of offline reinforcement learning pre-training and reduce the scale of data needed to be effective. Our approach leverages a world model of the environment trained on the offline dataset to augment states during offline pre-training. We evaluate our approach on a variety of MuJoCo robotic tasks and our results show it can jump-start online fine-tuning and substantially reduce - in some cases by an order of magnitude - the required number of environment interactions.