Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning
This work addresses the data bottleneck in offline RL for researchers and practitioners, offering a novel data generation method that can enhance performance without algorithmic changes, though it is incremental in its approach.
The paper tackles the problem of limited data diversity in offline reinforcement learning by proposing ExORL, a data-centric approach that generates exploratory data with unsupervised reward-free exploration and relabels it for downstream tasks. The result shows that vanilla off-policy RL algorithms using this data outperform or match state-of-the-art offline RL algorithms on downstream tasks.
Recent progress in deep learning has relied on access to large and diverse datasets. Such data-driven progress has been less evident in offline reinforcement learning (RL), because offline RL data is usually collected to optimize specific target tasks limiting the data's diversity. In this work, we propose Exploratory data for Offline RL (ExORL), a data-centric approach to offline RL. ExORL first generates data with unsupervised reward-free exploration, then relabels this data with a downstream reward before training a policy with offline RL. We find that exploratory data allows vanilla off-policy RL algorithms, without any offline-specific modifications, to outperform or match state-of-the-art offline RL algorithms on downstream tasks. Our findings suggest that data generation is as important as algorithmic advances for offline RL and hence requires careful consideration from the community. Code and data can be found at https://github.com/denisyarats/exorl .