Safe Wasserstein Constrained Deep Q-Learning
This work addresses safety-critical applications like battery management, though it is incremental as it builds on existing constrained MDP frameworks.
The paper tackles the problem of ensuring safety in online reinforcement learning by introducing a distributionally robust Q-Learning algorithm (DrQ) that uses Wasserstein ambiguity sets to provide probabilistic safety guarantees, resulting in improved safety compared to conventional methods in a lithium-ion battery fast charging case study.
This paper presents a distributionally robust Q-Learning algorithm (DrQ) which leverages Wasserstein ambiguity sets to provide idealistic probabilistic out-of-sample safety guarantees during online learning. First, we follow past work by separating the constraint functions from the principal objective to create a hierarchy of machines which estimate the feasible state-action space within the constrained Markov decision process (CMDP). DrQ works within this framework by augmenting constraint costs with tightening offset variables obtained through Wasserstein distributionally robust optimization (DRO). These offset variables correspond to worst-case distributions of modeling error characterized by the TD-errors of the constraint Q-functions. This procedure allows us to safely approach the nominal constraint boundaries. Using a case study of lithium-ion battery fast charging, we explore how idealistic safety guarantees translate to generally improved safety relative to conventional methods.