SURE-Val: Safe Urban Relevance Extension and Validation
This provides a validation framework for relevance definitions in urban automated driving perception systems, though it appears incremental as it extends existing highway methods.
The authors tackled the problem of insufficiently specified relevance definitions for perception evaluation in urban automated driving by extending an existing highway relevance method to urban domains and creating a novel validation method using motion prediction components. They successfully validated their relevance criteria by showing that removing irrelevant objects doesn't significantly affect prediction performance across a large-scale dataset.
To evaluate perception components of an automated driving system, it is necessary to define the relevant objects. While the urban domain is popular among perception datasets, relevance is insufficiently specified for this domain. Therefore, this work adopts an existing method to define relevance in the highway domain and expands it to the urban domain. While different conceptualizations and definitions of relevance are present in literature, there is a lack of methods to validate these definitions. Therefore, this work presents a novel relevance validation method leveraging a motion prediction component. The validation leverages the idea that removing irrelevant objects should not influence a prediction component which reflects human driving behavior. The influence on the prediction is quantified by considering the statistical distribution of prediction performance across a large-scale dataset. The validation procedure is verified using criteria specifically designed to exclude relevant objects. The validation method is successfully applied to the relevance criteria from this work, thus supporting their validity.