Efficient Determination of Safety Requirements for Perception Systems
This work addresses safety requirements for perception system designers in autonomy, though it is incremental as it builds on existing black-box estimation techniques.
The paper tackles the problem of efficiently determining safe performance characteristics for perception systems within autonomous systems, using a new method called smoothing bandits that improves accuracy and efficiency over baselines in a vision-based aircraft collision avoidance scenario.
Perception systems operate as a subcomponent of the general autonomy stack, and perception system designers often need to optimize performance characteristics while maintaining safety with respect to the overall closed-loop system. For this reason, it is useful to distill high-level safety requirements into component-level requirements on the perception system. In this work, we focus on efficiently determining sets of safe perception system performance characteristics given a black-box simulator of the fully-integrated, closed-loop system. We combine the advantages of common black-box estimation techniques such as Gaussian processes and threshold bandits to develop a new estimation method, which we call smoothing bandits. We demonstrate our method on a vision-based aircraft collision avoidance problem and show improvements in terms of both accuracy and efficiency over the Gaussian process and threshold bandit baselines.