Criticality Metrics for Relevance Classification in Safety Evaluation of Object Detection in Automated Driving
This addresses safety evaluation for automated vehicles, but appears incremental as it builds on existing metrics with new application methods.
The paper tackles the problem of evaluating object detection systems for automated driving safety by analyzing criticality metrics to distinguish relevant from non-relevant objects, demonstrating up to a 100% improvement in criticality classification accuracy through novel application strategies.
Ensuring safety is the primary objective of automated driving, which necessitates a comprehensive and accurate perception of the environment. While numerous performance evaluation metrics exist for assessing perception capabilities, incorporating safety-specific metrics is essential to reliably evaluate object detection systems. A key component for safety evaluation is the ability to distinguish between relevant and non-relevant objects - a challenge addressed by criticality or relevance metrics. This paper presents the first in-depth analysis of criticality metrics for safety evaluation of object detection systems. Through a comprehensive review of existing literature, we identify and assess a range of applicable metrics. Their effectiveness is empirically validated using the DeepAccident dataset, which features a variety of safety-critical scenarios. To enhance evaluation accuracy, we propose two novel application strategies: bidirectional criticality rating and multi-metric aggregation. Our approach demonstrates up to a 100% improvement in terms of criticality classification accuracy, highlighting its potential to significantly advance the safety evaluation of object detection systems in automated vehicles.