LET-3D-AP: Longitudinal Error Tolerant 3D Average Precision for Camera-Only 3D Detection
This addresses the evaluation challenge for camera-only 3D detection in autonomous driving, offering incremental improvements to metrics and benchmarking.
The paper tackles the problem of evaluating camera-only 3D detectors, which suffer from depth inaccuracies, by proposing new metrics (LET-3D-AP and LET-3D-APL) that tolerate longitudinal errors, and finds that state-of-the-art camera-based detectors can outperform LiDAR-based ones at 10% depth error tolerance.
The 3D Average Precision (3D AP) relies on the intersection over union between predictions and ground truth objects. However, camera-only detectors have limited depth accuracy, which may cause otherwise reasonable predictions that suffer from such longitudinal localization errors to be treated as false positives. We therefore propose variants of the 3D AP metric to be more permissive with respect to depth estimation errors. Specifically, our novel longitudinal error tolerant metrics, LET-3D-AP and LET-3D-APL, allow longitudinal localization errors of the prediction boxes up to a given tolerance. To evaluate the proposed metrics, we also construct a new test set for the Waymo Open Dataset, tailored to camera-only 3D detection methods. Surprisingly, we find that state-of-the-art camera-based detectors can outperform popular LiDAR-based detectors with our new metrics past at 10% depth error tolerance, suggesting that existing camera-based detectors already have the potential to surpass LiDAR-based detectors in downstream applications. We believe the proposed metrics and the new benchmark dataset will facilitate advances in the field of camera-only 3D detection by providing more informative signals that can better indicate the system-level performance.