Measuring Data Quality for Dataset Selection in Offline Reinforcement Learning
This addresses a practical dilemma for practitioners in offline RL, where dataset choice critically impacts algorithm performance, and it is incremental as it introduces new metrics for an underexplored problem.
The paper tackles the problem of selecting datasets for offline reinforcement learning by proposing three simple indicators—estimated relative return improvement (ERI), estimated action stochasticity (EAS), and their combination (COI)—and empirically shows they are effective for this task.
Recently developed offline reinforcement learning algorithms have made it possible to learn policies directly from pre-collected datasets, giving rise to a new dilemma for practitioners: Since the performance the algorithms are able to deliver depends greatly on the dataset that is presented to them, practitioners need to pick the right dataset among the available ones. This problem has so far not been discussed in the corresponding literature. We discuss ideas how to select promising datasets and propose three very simple indicators: Estimated relative return improvement (ERI) and estimated action stochasticity (EAS), as well as a combination of the two (COI), and empirically show that despite their simplicity they can be very effectively used for dataset selection.