Safe Policy Improvement with Baseline Bootstrapping
This addresses the challenge of reliably improving policies from fixed datasets without online interaction, which is crucial for applications where data collection is costly or risky, though it appears incremental as it builds on existing SPI and bootstrapping ideas.
This paper tackles the problem of Safe Policy Improvement (SPI) in Batch Reinforcement Learning by proposing SPIBB, which bootstraps the trained policy with a baseline when uncertainty is high, ensuring performance at least as good as the baseline policy. The results show superiority in safety and mean performance on random MDPs and demonstrate efficient training from batch data without environment interaction, including a model-free version called SPIBB-DQN.
This paper considers Safe Policy Improvement (SPI) in Batch Reinforcement Learning (Batch RL): from a fixed dataset and without direct access to the true environment, train a policy that is guaranteed to perform at least as well as the baseline policy used to collect the data. Our approach, called SPI with Baseline Bootstrapping (SPIBB), is inspired by the knows-what-it-knows paradigm: it bootstraps the trained policy with the baseline when the uncertainty is high. Our first algorithm, $Π_b$-SPIBB, comes with SPI theoretical guarantees. We also implement a variant, $Π_{\leq b}$-SPIBB, that is even more efficient in practice. We apply our algorithms to a motivational stochastic gridworld domain and further demonstrate on randomly generated MDPs the superiority of SPIBB with respect to existing algorithms, not only in safety but also in mean performance. Finally, we implement a model-free version of SPIBB and show its benefits on a navigation task with deep RL implementation called SPIBB-DQN, which is, to the best of our knowledge, the first RL algorithm relying on a neural network representation able to train efficiently and reliably from batch data, without any interaction with the environment.