Counterfactually Safe Reinforcement Learning
For safety-critical RL applications, this work provides a principled method to mitigate individual harm while maintaining performance.
The paper formalizes individual harm from a counterfactual perspective and proposes a two-stage procedure for learning policies that maximize expected return while controlling harm. Experiments on simulated and real-world data show the approach effectively controls harm rates.
Reinforcement learning algorithms are generally designed to maximize the expected return across a population. However, a policy that is optimal on average may be suboptimal for certain individuals, leading to potential safety concerns. To address this, we first formalize the notion of individual harm from a counterfactual perspective and define harm as the event in which a chosen action results in a strictly worse outcome than a baseline alternative. We then propose a general two-stage procedure for learning policies that maximize the expected return while accounting for individual harm. We further establish the finite-sample properties of the learned policy, derive an upper bound on its sub-optimality gap, and show that the harm rate remains well-controlled. Numerical experiments on both simulated and real-world datasets demonstrate the effectiveness of the proposed approach.