On the Need for a Statistical Foundation in Scenario-Based Testing of Autonomous Vehicles
This addresses safety assurance challenges for autonomous vehicle developers, but it is incremental as it builds on existing software testing methods.
The paper tackles the lack of statistical rigor in scenario-based testing for autonomous vehicles, proposing models to quantify failure probabilities and showing that neither scenario-based nor mile-based testing is universally superior.
Scenario-based testing has emerged as a common method for autonomous vehicles (AVs) safety assessment, offering a more efficient alternative to mile-based testing by focusing on high-risk scenarios. However, fundamental questions persist regarding its stopping rules, residual risk estimation, debug effectiveness, and the impact of simulation fidelity on safety claims. This paper argues that a rigorous statistical foundation is essential to address these challenges and enable rigorous safety assurance. By drawing parallels between AV testing and established software testing methods, we identify shared research gaps and reusable solutions. We propose proof-of-concept models to quantify the probability of failure per scenario (\textit{pfs}) and evaluate testing effectiveness under varying conditions. Our analysis reveals that neither scenario-based nor mile-based testing universally outperforms the other. Furthermore, we give an example of formal reasoning about alignment of synthetic and real-world testing outcomes, a first step towards supporting statistically defensible simulation-based safety claims.