Evaluation of Out-of-Distribution Detection Performance on Autonomous Driving Datasets
This work addresses safety verification for DNNs in automotive perception, though it is incremental as it applies an existing method to new data.
The paper tackled the problem of evaluating out-of-distribution detection for semantic segmentation DNNs in autonomous driving, finding that using a Mahalanobis distance-based score can drastically reduce classification risk at the cost of pixel coverage across multiple automotive datasets.
Safety measures need to be systemically investigated to what extent they evaluate the intended performance of Deep Neural Networks (DNNs) for critical applications. Due to a lack of verification methods for high-dimensional DNNs, a trade-off is needed between accepted performance and handling of out-of-distribution (OOD) samples. This work evaluates rejecting outputs from semantic segmentation DNNs by applying a Mahalanobis distance (MD) based on the most probable class-conditional Gaussian distribution for the predicted class as an OOD score. The evaluation follows three DNNs trained on the Cityscapes dataset and tested on four automotive datasets and finds that classification risk can drastically be reduced at the cost of pixel coverage, even when applied on unseen datasets. The applicability of our findings will support legitimizing safety measures and motivate their usage when arguing for safe usage of DNNs in automotive perception.