RayD3D: Distilling Depth Knowledge Along the Ray for Robust Multi-View 3D Object Detection
This work improves the robustness of multi-view 3D object detection for autonomous driving, which is an incremental gain for the field.
This paper addresses the challenge of inaccurate depth prediction in multi-view 3D object detection for autonomous driving by proposing RayD3D, a method that distills depth knowledge along the ray from LiDAR to camera models. RayD3D significantly improves the robustness of three representative BEV-based models (BEVDet, BEVDepth4D, and BEVFormer) on both clean NuScenes and corrupted RoboBEV datasets without increasing inference costs.
Multi-view 3D detection with bird's eye view (BEV) is crucial for autonomous driving and robotics, but its robustness in real-world is limited as it struggles to predict accurate depth values. A mainstream solution, cross-modal distillation, transfers depth information from LiDAR to camera models but also unintentionally transfers depth-irrelevant information (e.g. LiDAR density). To mitigate this issue, we propose RayD3D, which transfers crucial depth knowledge along the ray: a line projecting from the camera to true location of an object. It is based on the fundamental imaging principle that predicted location of this object can only vary along this ray, which is finally determined by predicted depth value. Therefore, distilling along the ray enables more effective depth information transfer. More specifically, we design two ray-based distillation modules. Ray-based Contrastive Distillation (RCD) incorporates contrastive learning into distillation by sampling along the ray to learn how LiDAR accurately locates objects. Ray-based Weighted Distillation (RWD) adaptively adjusts distillation weight based on the ray to minimize the interference of depth-irrelevant information in LiDAR. For validation, we widely apply RayD3D into three representative types of BEV-based models, including BEVDet, BEVDepth4D, and BEVFormer. Our method is trained on clean NuScenes, and tested on both clean NuScenes and RoboBEV with a variety types of data corruptions. Our method significantly improves the robustness of all the three base models in all scenarios without increasing inference costs, and achieves the best when compared to recently released multi-view and distillation models.