Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning
This work addresses safety guarantees in reinforcement learning for applications requiring high-probability safety, representing a novel method rather than an incremental improvement.
The paper tackled the problem of maintaining safety constraints with high probability in reinforcement learning by moving from expectation-form constraints to quantile-constrained RL, resulting in a model that fully meets safety requirements while outperforming state-of-the-art benchmarks with higher return.
Safe reinforcement learning (RL) is a popular and versatile paradigm to learn reward-maximizing policies with safety guarantees. Previous works tend to express the safety constraints in an expectation form due to the ease of implementation, but this turns out to be ineffective in maintaining safety constraints with high probability. To this end, we move to the quantile-constrained RL that enables a higher level of safety without any expectation-form approximations. We directly estimate the quantile gradients through sampling and provide the theoretical proofs of convergence. Then a tilted update strategy for quantile gradients is implemented to compensate the asymmetric distributional density, with a direct benefit of return performance. Experiments demonstrate that the proposed model fully meets safety requirements (quantile constraints) while outperforming the state-of-the-art benchmarks with higher return.