Guiding Safe Exploration with Weakest Preconditions
This work addresses safety constraints during training in reinforcement learning, which is crucial for safety-critical applications, though it appears incremental as it builds on existing safe learning techniques.
The paper tackled the problem of safe exploration in reinforcement learning for safety-critical settings by introducing SPICE, a neurosymbolic approach that uses symbolic weakest preconditions for online shielding, resulting in fewer safety violations while achieving comparable performance to existing techniques.
In reinforcement learning for safety-critical settings, it is often desirable for the agent to obey safety constraints at all points in time, including during training. We present a novel neurosymbolic approach called SPICE to solve this safe exploration problem. SPICE uses an online shielding layer based on symbolic weakest preconditions to achieve a more precise safety analysis than existing tools without unduly impacting the training process. We evaluate the approach on a suite of continuous control benchmarks and show that it can achieve comparable performance to existing safe learning techniques while incurring fewer safety violations. Additionally, we present theoretical results showing that SPICE converges to the optimal safe policy under reasonable assumptions.