An Empirical Evaluation of Posterior Sampling for Constrained Reinforcement Learning
This work addresses constrained reinforcement learning, offering more efficient and simpler algorithms for researchers and practitioners, though it appears incremental as it builds on existing formulations.
The paper tackled the problem of efficient exploration in constrained reinforcement learning by proposing two simple posterior sampling algorithms, which achieved state-of-the-art performance and sometimes significantly outperformed optimistic algorithms.
We study a posterior sampling approach to efficient exploration in constrained reinforcement learning. Alternatively to existing algorithms, we propose two simple algorithms that are more efficient statistically, simpler to implement and computationally cheaper. The first algorithm is based on a linear formulation of CMDP, and the second algorithm leverages the saddle-point formulation of CMDP. Our empirical results demonstrate that, despite its simplicity, posterior sampling achieves state-of-the-art performance and, in some cases, significantly outperforms optimistic algorithms.