More for Less: Safe Policy Improvement With Stronger Performance Guarantees
This work addresses the sample efficiency challenge in offline reinforcement learning for practitioners, though it is incremental as it builds upon existing SPI methods like SPIBB.
The paper tackles the problem of requiring large sample sizes for safe policy improvement (SPI) in offline reinforcement learning by introducing a novel approach that uses implicit transformations to derive tighter performance guarantees, resulting in a significant reduction in sample complexity for the SPIBB algorithm as shown in empirical evaluations on standard benchmarks.
In an offline reinforcement learning setting, the safe policy improvement (SPI) problem aims to improve the performance of a behavior policy according to which sample data has been generated. State-of-the-art approaches to SPI require a high number of samples to provide practical probabilistic guarantees on the improved policy's performance. We present a novel approach to the SPI problem that provides the means to require less data for such guarantees. Specifically, to prove the correctness of these guarantees, we devise implicit transformations on the data set and the underlying environment model that serve as theoretical foundations to derive tighter improvement bounds for SPI. Our empirical evaluation, using the well-established SPI with baseline bootstrapping (SPIBB) algorithm, on standard benchmarks shows that our method indeed significantly reduces the sample complexity of the SPIBB algorithm.