Plane and Sample: Maximizing Information about Autonomous Vehicle Performance using Submodular Optimization
This work addresses the need for systematic and scalable evaluation of autonomous vehicles, which is incremental as it builds on existing sampling techniques with a novel optimization approach.
The paper tackles the problem of efficiently evaluating autonomous vehicle performance across different operational domains and functionalities by reformulating scenario sampling as a submodular optimization problem, resulting in a 23% improvement over Latin Hypercube Sampling by requiring only 7.5% of the scenario space to meet a stopping criterion.
As autonomous vehicles (AVs) take on growing Operational Design Domains (ODDs), they need to go through a systematic, transparent, and scalable evaluation process to demonstrate their benefits to society. Current scenario sampling techniques for AV performance evaluation usually focus on a specific functionality, such as lane changing, and do not accommodate a transfer of information about an AV system from one ODD to the next. In this paper, we reformulate the scenario sampling problem across ODDs and functionalities as a submodular optimization problem. To do so, we abstract AV performance as a Bayesian Hierarchical Model, which we use to infer information gained by revealing performance in new scenarios. We propose the information gain as a measure of scenario relevance and evaluation progress. Furthermore, we leverage the submodularity, or diminishing returns, property of the information gain not only to find a near-optimal scenario set, but also to propose a stopping criterion for an AV performance evaluation campaign. We find that we only need to explore about 7.5% of the scenario space to meet this criterion, a 23% improvement over Latin Hypercube Sampling.