Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation
This addresses the safety and efficiency challenges in AV testing for the public and industry, though it is incremental as it builds on existing simulation and sampling methods.
The paper tackles the problem of scalable and rigorous testing for autonomous vehicles by implementing a simulation framework that uses adaptive importance-sampling to accelerate rare-event probability evaluation, demonstrating speed-ups of 2-20 times over naive Monte Carlo and 10-300P times over real-world testing in a highway scenario.
While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the $\textit{de facto}$ evaluation environment, places the public in danger, and, due to the rare nature of accidents, will require billions of miles in order to statistically validate performance claims. We implement a simulation framework that can test an entire modern autonomous driving system, including, in particular, systems that employ deep-learning perception and control algorithms. Using adaptive importance-sampling methods to accelerate rare-event probability evaluation, we estimate the probability of an accident under a base distribution governing standard traffic behavior. We demonstrate our framework on a highway scenario, accelerating system evaluation by $2$-$20$ times over naive Monte Carlo sampling methods and $10$-$300 \mathsf{P}$ times (where $\mathsf{P}$ is the number of processors) over real-world testing.