A Scenario-Based Platform for Testing Autonomous Vehicle Behavior Prediction Models in Simulation
This work addresses the problem of ensuring safety for autonomous vehicles by enabling more comprehensive testing of prediction models, though it is incremental as it builds on existing simulation and testing methods.
The paper tackles the challenge of testing autonomous vehicle behavior prediction models by presenting a simulation-based platform that supports intuitive scenario modeling, multi-objective evaluation, and scalable parallel testing, demonstrating its effectiveness by finding failure scenarios for a trained model.
Behavior prediction remains one of the most challenging tasks in the autonomous vehicle (AV) software stack. Forecasting the future trajectories of nearby agents plays a critical role in ensuring road safety, as it equips AVs with the necessary information to plan safe routes of travel. However, these prediction models are data-driven and trained on data collected in real life that may not represent the full range of scenarios an AV can encounter. Hence, it is important that these prediction models are extensively tested in various test scenarios involving interactive behaviors prior to deployment. To support this need, we present a simulation-based testing platform which supports (1) intuitive scenario modeling with a probabilistic programming language called Scenic, (2) specifying a multi-objective evaluation metric with a partial priority ordering, (3) falsification of the provided metric, and (4) parallelization of simulations for scalable testing. As a part of the platform, we provide a library of 25 Scenic programs that model challenging test scenarios involving interactive traffic participant behaviors. We demonstrate the effectiveness and the scalability of our platform by testing a trained behavior prediction model and searching for failure scenarios.