Probabilistic Metamodels for an Efficient Characterization of Complex Driving Scenarios
This work addresses the safety-critical need for efficient scenario-based testing in automated vehicles, though it is incremental in improving existing metamodel approaches.
The paper tackled the problem of efficiently characterizing complex driving scenarios for automated vehicle safety validation by analyzing the predictive performance of various metamodels and introducing an iterative test case selection approach, finding that appropriate test case selection is more important than metamodel choice, with Bayesian neural networks benefiting from large data and Gaussian processes offering higher reliability.
To validate the safety of automated vehicles (AV), scenario-based testing aims to systematically describe driving scenarios an AV might encounter. In this process, continuous inputs such as velocities result in an infinite number of possible variations of a scenario. Thus, metamodels are used to perform analyses or to select specific variations for examination. However, despite the safety criticality of AV testing, metamodels are usually seen as a part of an overall approach, and their predictions are not questioned. This paper analyzes the predictive performance of Gaussian processes (GP), deep Gaussian processes, extra-trees, and Bayesian neural networks (BNN), considering four scenarios with 5 to 20 inputs. Building on this, an iterative approach is introduced and evaluated, which allows to efficiently select test cases for common analysis tasks. The results show that regarding predictive performance, the appropriate selection of test cases is more important than the choice of metamodels. However, the choice of metamodels remains crucial: Their great flexibility allows BNNs to benefit from large amounts of data and to model even the most complex scenarios. In contrast, less flexible models like GPs convince with higher reliability. Hence, relevant test cases are best explored using scalable virtual test setups and flexible models. Subsequently, more realistic test setups and more reliable models can be used for targeted testing and validation.