In the Driver's Seat: A Multi-Company Study on the Reality of Autonomous Driving System Testing
For researchers and practitioners in autonomous driving, this study provides a comprehensive, industry-grounded overview of testing practices and challenges, but it is an incremental qualitative analysis rather than a novel technical solution.
This interview study with experts from nine companies across six countries reveals that current autonomous driving system testing practices rely on scenario-based and X-in-the-loop approaches, with major challenges in scenario realism, coverage, simulation fidelity, and acceptance criteria. The study proposes an evidence-centered closed-loop testing framework to guide future testing.
Autonomous driving systems (ADS) are rapidly advancing and increasingly deployed in real-world applications. This creates growing demands for effective testing to ensure system functionality and safety. However, ADS testing remains complex and lacks well-established standards for scenario selection, performance evaluation, and acceptance criteria. To better understand current ADS testing practices and challenges, we conducted an interview study with experts working on ADS development and testing in nine companies from six different countries. Through thematic analysis, we synthesized industrial testing practices, challenges, potential solutions, future trends, and proposed an evidence-centered closed-loop testing framework for ADS testing. Our findings show that current practices primarily focus on scenario-based and X-in-the-loop testing approaches, supported by diverse tools, metrics, benchmarks, and testing strategies. The participants highlighted major challenges related to scenario realism, scenario coverage, simulation fidelity, and acceptance criteria, while also discussing potential solutions such as the use of AI, world models, and end-to-end approaches. Furthermore, participants envisioned future ADS testing to become more automated, data-driven, and transparent across the industry. Overall, this study provides a comprehensive industry-grounded overview of ADS testing, proposes an evidence-centered closed-loop testing framework to provide actionable guidance for ADS testing, and outlines important directions for future research and practice.