Evaluation Uncertainty in Data-Driven Self-Driving Testing
This work addresses uncertainty in safety-critical evaluations for self-driving technology, but it is incremental as it builds on existing Monte Carlo and bootstrap methods.
The paper tackles the problem of uncertainty in safety evaluation for self-driving technologies by proposing a method to assess the impact of data variability on probability estimates, demonstrating it as a diagnostic tool for input model quality in autonomous vehicle safety evaluation.
Safety evaluation of self-driving technologies has been extensively studied. One recent approach uses Monte Carlo based evaluation to estimate the occurrence probabilities of safety-critical events as safety measures. These Monte Carlo samples are generated from stochastic input models constructed based on real-world data. In this paper, we propose an approach to assess the impact on the probability estimates from the evaluation procedures due to the estimation error caused by data variability. Our proposed method merges the classical bootstrap method for estimating input uncertainty with a likelihood ratio based scheme to reuse experiment outputs. This approach is economical and efficient in terms of implementation costs in assessing input uncertainty for the evaluation of self-driving technology. We use an example in autonomous vehicle (AV) safety evaluation to demonstrate the proposed approach as a diagnostic tool for the quality of the fitted input model.