Safe MDP Planning by Learning Temporal Patterns of Undesirable Trajectories and Averting Negative Side Effects
This addresses safety in reinforcement learning for real-world applications where incomplete models pose risks, though it appears incremental as it builds on existing safe MDP planning with trajectory-based extensions.
The paper tackles the problem of safe MDP planning when state representations lack fidelity to specify safety constraints, which can cause unintended negative side effects. The result is an approach that learns non-Markovian safety patterns from trajectory labels and incorporates them via a Lagrange multiplier method, showing it satisfies complex safety constraints while optimizing returns and outperforms previous methods for Markovian negative side effects.
In safe MDP planning, a cost function based on the current state and action is often used to specify safety aspects. In the real world, often the state representation used may lack sufficient fidelity to specify such safety constraints. Operating based on an incomplete model can often produce unintended negative side effects (NSEs). To address these challenges, first, we associate safety signals with state-action trajectories (rather than just an immediate state-action). This makes our safety model highly general. We also assume categorical safety labels are given for different trajectories, rather than a numerical cost function, which is harder to specify by the problem designer. We then employ a supervised learning model to learn such non-Markovian safety patterns. Second, we develop a Lagrange multiplier method, which incorporates the safety model and the underlying MDP model in a single computation graph to facilitate agent learning of safe behaviors. Finally, our empirical results on a variety of discrete and continuous domains show that this approach can satisfy complex non-Markovian safety constraints while optimizing an agent's total returns, is highly scalable, and is also better than the previous best approach for Markovian NSEs.