Closed-Loop Policies for Operational Tests of Safety-Critical Systems
This work addresses strategic resource allocation for manufacturers and regulators in safety-critical domains like autonomous vehicles, offering incremental improvements in test policy optimization.
The paper tackles the problem of optimizing test scheduling for safety-critical systems by framing it as a Markov decision process, resulting in efficient closed-loop policies that provide guidance on when to continue testing or adjust requirements.
Manufacturers of safety-critical systems must make the case that their product is sufficiently safe for public deployment. Much of this case often relies upon critical event outcomes from real-world testing, requiring manufacturers to be strategic about how they allocate testing resources in order to maximize their chances of demonstrating system safety. This work frames the partially observable and belief-dependent problem of test scheduling as a Markov decision process, which can be solved efficiently to yield closed-loop manufacturer testing policies. By solving for policies over a wide range of problem formulations, we are able to provide high-level guidance for manufacturers and regulators on issues relating to the testing of safety-critical systems. This guidance spans an array of topics, including circumstances under which manufacturers should continue testing despite observed incidents, when manufacturers should test aggressively, and when regulators should increase or reduce the real-world testing requirements for an autonomous vehicle.