Evaluating the Impact of Flaky Simulators on Testing Autonomous Driving Systems
This addresses the reliability of simulation-based testing for autonomous driving systems, which is crucial for developers and safety engineers, though it is incremental as it applies existing ML methods to a specific domain problem.
The study tackled the problem of flaky simulators causing inconsistent test results in autonomous driving systems, finding that flakiness is common and significantly impacts randomized testing, and that machine learning classifiers can identify flaky tests with F1-scores up to 96% using only a single run.
Simulators are widely used to test Autonomous Driving Systems (ADS), but their potential flakiness can lead to inconsistent test results. We investigate test flakiness in simulation-based testing of ADS by addressing two key questions: (1) How do flaky ADS simulations impact automated testing that relies on randomized algorithms? and (2) Can machine learning (ML) effectively identify flaky ADS tests while decreasing the required number of test reruns? Our empirical results, obtained from two widely-used open-source ADS simulators and five diverse ADS test setups, show that test flakiness in ADS is a common occurrence and can significantly impact the test results obtained by randomized algorithms. Further, our ML classifiers effectively identify flaky ADS tests using only a single test run, achieving F1-scores of $85$%, $82$% and $96$% for three different ADS test setups. Our classifiers significantly outperform our non-ML baseline, which requires executing tests at least twice, by $31$%, $21$%, and $13$% in F1-score performance, respectively. We conclude with a discussion on the scope, implications and limitations of our study. We provide our complete replication package in a Github repository.