Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learning
This work addresses the challenge of applying theoretically motivated offline RL algorithms to more complex, non-tabular settings, which is an incremental advancement for the reinforcement learning community.
The paper tackled the problem of scaling offline reinforcement learning to larger state and action spaces by developing deep-SPIBB, a method that incorporates scalable uncertainty estimates from deep learning. The result showed that deep-SPIBB outperformed pessimism-based approaches with the same uncertainty estimates and performed competitively with other baselines across multiple environments and datasets.
Most theoretically motivated work in the offline reinforcement learning setting requires precise uncertainty estimates. This requirement restricts the algorithms derived in that work to the tabular and linear settings where such estimates exist. In this work, we develop a novel method for incorporating scalable uncertainty estimates into an offline reinforcement learning algorithm called deep-SPIBB that extends the SPIBB family of algorithms to environments with larger state and action spaces. We use recent innovations in uncertainty estimation from the deep learning community to get more scalable uncertainty estimates to plug into deep-SPIBB. While these uncertainty estimates do not allow for the same theoretical guarantees as in the tabular case, we argue that the SPIBB mechanism for incorporating uncertainty is more robust and flexible than pessimistic approaches that incorporate the uncertainty as a value function penalty. We bear this out empirically, showing that deep-SPIBB outperforms pessimism based approaches with access to the same uncertainty estimates and performs at least on par with a variety of other strong baselines across several environments and datasets.