Physics-informed RL for Maximal Safety Probability Estimation
This addresses the challenge of safe control and learning in robotics or autonomous systems by enabling risk estimation without extensive data coverage, though it appears incremental as it builds on standard RL methods with physics constraints.
The paper tackles the problem of estimating long-term safety probability for safe control and learning, which is costly due to rare events and risky states, by proposing a Physics-Informed Reinforcement Learning (PIRL) algorithm that converts multiplicative safety probabilities into additive costs and uses physics constraints to learn efficiently with sparse rewards and short-term samples.
Accurate risk quantification and reachability analysis are crucial for safe control and learning, but sampling from rare events, risky states, or long-term trajectories can be prohibitively costly. Motivated by this, we study how to estimate the long-term safety probability of maximally safe actions without sufficient coverage of samples from risky states and long-term trajectories. The use of maximal safety probability in control and learning is expected to avoid conservative behaviors due to over-approximation of risk. Here, we first show that long-term safety probability, which is multiplicative in time, can be converted into additive costs and be solved using standard reinforcement learning methods. We then derive this probability as solutions of partial differential equations (PDEs) and propose Physics-Informed Reinforcement Learning (PIRL) algorithm. The proposed method can learn using sparse rewards because the physics constraints help propagate risk information through neighbors. This suggests that, for the purpose of extracting more information for efficient learning, physics constraints can serve as an alternative to reward shaping. The proposed method can also estimate long-term risk using short-term samples and deduce the risk of unsampled states. This feature is in stark contrast with the unconstrained deep RL that demands sufficient data coverage. These merits of the proposed method are demonstrated in numerical simulation.