Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data
This work addresses the need for systematic and data-efficient algorithm selection in offline RL, which is crucial for practitioners in fields like healthcare, education, and robotics, though it is incremental as it adapts existing supervised learning methods to this setting.
The authors tackled the problem of selecting the best algorithm and hyperparameters for offline reinforcement learning when only limited historical data is available, by introducing a task-agnostic pipeline that uses multiple data splits to improve reliability, resulting in higher-performing policies across various simulation domains.
Offline reinforcement learning (RL) can be used to improve future performance by leveraging historical data. There exist many different algorithms for offline RL, and it is well recognized that these algorithms, and their hyperparameter settings, can lead to decision policies with substantially differing performance. This prompts the need for pipelines that allow practitioners to systematically perform algorithm-hyperparameter selection for their setting. Critically, in most real-world settings, this pipeline must only involve the use of historical data. Inspired by statistical model selection methods for supervised learning, we introduce a task- and method-agnostic pipeline for automatically training, comparing, selecting, and deploying the best policy when the provided dataset is limited in size. In particular, our work highlights the importance of performing multiple data splits to produce more reliable algorithm-hyperparameter selection. While this is a common approach in supervised learning, to our knowledge, this has not been discussed in detail in the offline RL setting. We show it can have substantial impacts when the dataset is small. Compared to alternate approaches, our proposed pipeline outputs higher-performing deployed policies from a broad range of offline policy learning algorithms and across various simulation domains in healthcare, education, and robotics. This work contributes toward the development of a general-purpose meta-algorithm for automatic algorithm-hyperparameter selection for offline RL.