Surprise Potential as a Measure of Interactivity in Driving Scenarios
This work addresses the challenge of curating datasets for more accurate autonomous vehicle performance assessment, though it is incremental as it builds on existing metrics and datasets.
The paper tackles the problem of identifying interactive scenarios in real-world driving logs for autonomous vehicle validation by introducing a novel metric based on surprise potential, achieving a correlation of over 0.82 with human-aligned preferences and outperforming existing methods.
Validating the safety and performance of an autonomous vehicle (AV) requires benchmarking on real-world driving logs. However, typical driving logs contain mostly uneventful scenarios with minimal interactions between road users. Identifying interactive scenarios in real-world driving logs enables the curation of datasets that amplify critical signals and provide a more accurate assessment of an AV's performance. In this paper, we present a novel metric that identifies interactive scenarios by measuring an AV's surprise potential on others. First, we identify three dimensions of the design space to describe a family of surprise potential measures. Second, we exhaustively evaluate and compare different instantiations of the surprise potential measure within this design space on the nuScenes dataset. To determine how well a surprise potential measure correctly identifies an interactive scenario, we use a reward model learned from human preferences to assess alignment with human intuition. Our proposed surprise potential, arising from this exhaustive comparative study, achieves a correlation of more than 0.82 with the human-aligned reward function, outperforming existing approaches. Lastly, we validate motion planners on curated interactive scenarios to demonstrate downstream applications.