SteinGate: Tail-Sensitive Safe Reinforcement Learning via Stein Discrepancy
For safe RL practitioners, SteinGate provides a non-parametric method to handle tail risks that standard expected-cost constraints miss, improving safety during training.
SteinGate introduces a distributional safety certificate using Kernelized Stein Discrepancy to detect and mitigate rare catastrophic cost tail events in safe RL, significantly reducing constraint violation frequency and severity while maintaining competitive returns.
Safe reinforcement learning typically enforces safety by bounding expected cumulative costs, a criterion that often fails to detect rare but catastrophic tail events. To overcome these limitations, this paper introduces SteinGate, a boundary-aware distributional safety certificate that replaces fragile tail fitting with a robust consistency check using Kernelized Stein Discrepancy while accounting for boundary atoms induced by clipped costs. SteinGate evaluates whether observed policy rollout costs remain consistent with a safe reference distribution, providing a non-parametric safety certificate. This certificate is used to dynamically adapt the learning regime: favoring reward-improving policy updates when rollouts remain consistent with the safe reference and switching to recovery behavior when the cost tail deviates. Experiments on continuous-control benchmarks demonstrate that SteinGate significantly reduces both the frequency and severity of constraint violations during training while maintaining competitive returns relative to state-of-the-art baselines.