On Offline Evaluation of 3D Object Detection for Autonomous Driving
This work addresses the problem of evaluating 3D object detection models for autonomous driving, providing empirical insights for researchers and practitioners, though it is incremental in refining evaluation practices.
The study investigated the correlation between offline 3D object detection metrics and actual driving performance in autonomous vehicles, finding that the nuScenes Detection Score is more predictive than average precision and cautioning against over-reliance on planner-centric metrics.
Prior work in 3D object detection evaluates models using offline metrics like average precision since closed-loop online evaluation on the downstream driving task is costly. However, it is unclear how indicative offline results are of driving performance. In this work, we perform the first empirical evaluation measuring how predictive different detection metrics are of driving performance when detectors are integrated into a full self-driving stack. We conduct extensive experiments on urban driving in the CARLA simulator using 16 object detection models. We find that the nuScenes Detection Score has a higher correlation to driving performance than the widely used average precision metric. In addition, our results call for caution on the exclusive reliance on the emerging class of `planner-centric' metrics.