Conformal Safety Shielding for Imperfect-Perception Agents
This addresses safety concerns for autonomous systems using learned perception components, though it is incremental as it builds on existing shield constructions.
The paper tackles the problem of ensuring safe control for autonomous agents with imperfect perception by proposing a shield that restricts actions based on state estimates, using conformal prediction to guarantee inclusion of the actual state with a user-specified probability, and demonstrates this with a case study on an autonomous airplane taxiing system.
We consider the problem of safe control in discrete autonomous agents that use learned components for imperfect perception (or more generally, state estimation) from high-dimensional observations. We propose a shield construction that provides run-time safety guarantees under perception errors by restricting the actions available to an agent, modeled as a Markov decision process, as a function of the state estimates. Our construction uses conformal prediction for the perception component, which guarantees that for each observation, the predicted set of estimates includes the actual state with a user-specified probability. The shield allows an action only if it is allowed for all the estimates in the predicted set, resulting in local safety. We also articulate and prove a global safety property of existing shield constructions for perfect-perception agents bounding the probability of reaching unsafe states if the agent always chooses actions prescribed by the shield. We illustrate our approach with a case-study of an experimental autonomous system that guides airplanes on taxiways using high-dimensional perception DNNs.