Quantitative Projection Coverage for Testing ML-enabled Autonomous Systems
This addresses the problem of safety justification for autonomous vehicles, but it is incremental as it builds on existing coverage metrics with a new quantitative approach.
The paper tackles the challenge of systematically testing neural network models for autonomous vehicles by proposing quantitative k-projection coverage as a metric to manage combinatorial explosion and guide data sampling, demonstrating feasibility through a prototype for a visual object detection unit.
Systematically testing models learned from neural networks remains a crucial unsolved barrier to successfully justify safety for autonomous vehicles engineered using data-driven approach. We propose quantitative k-projection coverage as a metric to mediate combinatorial explosion while guiding the data sampling process. By assuming that domain experts propose largely independent environment conditions and by associating elements in each condition with weights, the product of these conditions forms scenarios, and one may interpret weights associated with each equivalence class as relative importance. Achieving full k-projection coverage requires that the data set, when being projected to the hyperplane formed by arbitrarily selected k-conditions, covers each class with number of data points no less than the associated weight. For the general case where scenario composition is constrained by rules, precisely computing k-projection coverage remains in NP. In terms of finding minimum test cases to achieve full coverage, we present theoretical complexity for important sub-cases and an encoding to 0-1 integer programming. We have implemented a research prototype that generates test cases for a visual object defection unit in automated driving, demonstrating the technological feasibility of our proposed coverage criterion.