Safe Explicable Policy Search
This work addresses the challenge of safe and explicable behavior generation in human-AI teaming, particularly for robotic applications, but it is incremental as it builds upon existing methods like Constrained Policy Optimization and Explicable Policy Search.
The paper tackles the problem of AI agents generating behaviors that meet user expectations while ensuring safety, by introducing Safe Explicable Policy Search (SEPS) as a constrained optimization approach that maximizes explicability under safety constraints, and demonstrates its efficacy in safety-gym environments and a physical robot experiment.
When users work with AI agents, they form conscious or subconscious expectations of them. Meeting user expectations is crucial for such agents to engage in successful interactions and teaming. However, users may form expectations of an agent that differ from the agent's planned behaviors. These differences lead to the consideration of two separate decision models in the planning process to generate explicable behaviors. However, little has been done to incorporate safety considerations, especially in a learning setting. We present Safe Explicable Policy Search (SEPS), which aims to provide a learning approach to explicable behavior generation while minimizing the safety risk, both during and after learning. We formulate SEPS as a constrained optimization problem where the agent aims to maximize an explicability score subject to constraints on safety and a suboptimality criterion based on the agent's model. SEPS innovatively combines the capabilities of Constrained Policy Optimization and Explicable Policy Search to introduce the capability of generating safe explicable behaviors to domains with continuous state and action spaces, which is critical for robotic applications. We evaluate SEPS in safety-gym environments and with a physical robot experiment to show its efficacy and relevance in human-AI teaming.