Safe Policy Improvement Approaches and their Limitations
This addresses safety-critical offline reinforcement learning by exposing limitations in existing methods and offering improved algorithms, though it is incremental as it builds on prior SPI work.
The paper identifies that Soft-SPIBB algorithms in Safe Policy Improvement (SPI) for offline reinforcement learning lack provable safety guarantees, and proposes Adv-Soft-SPIBB adaptations that are provably safe, with Lower-Approx-Soft-SPIBB achieving the best performance in benchmarks.
Safe Policy Improvement (SPI) is an important technique for offline reinforcement learning in safety critical applications as it improves the behavior policy with a high probability. We classify various SPI approaches from the literature into two groups, based on how they utilize the uncertainty of state-action pairs. Focusing on the Soft-SPIBB (Safe Policy Improvement with Soft Baseline Bootstrapping) algorithms, we show that their claim of being provably safe does not hold. Based on this finding, we develop adaptations, the Adv-Soft-SPIBB algorithms, and show that they are provably safe. A heuristic adaptation, Lower-Approx-Soft-SPIBB, yields the best performance among all SPIBB algorithms in extensive experiments on two benchmarks. We also check the safety guarantees of the provably safe algorithms and show that huge amounts of data are necessary such that the safety bounds become useful in practice.