Radar-Camera Pixel Depth Association for Depth Completion
This work addresses a domain-specific challenge in autonomous driving by improving depth completion for radar-camera fusion, representing an incremental advance over existing methods.
The paper tackles the problem of poor pixel-level association between sparse automotive radar and camera data for depth completion, proposing a radar-to-pixel mapping stage that densifies radar returns and enables superior performance compared to using camera or radar alone on the nuScenes dataset.
While radar and video data can be readily fused at the detection level, fusing them at the pixel level is potentially more beneficial. This is also more challenging in part due to the sparsity of radar, but also because automotive radar beams are much wider than a typical pixel combined with a large baseline between camera and radar, which results in poor association between radar pixels and color pixel. A consequence is that depth completion methods designed for LiDAR and video fare poorly for radar and video. Here we propose a radar-to-pixel association stage which learns a mapping from radar returns to pixels. This mapping also serves to densify radar returns. Using this as a first stage, followed by a more traditional depth completion method, we are able to achieve image-guided depth completion with radar and video. We demonstrate performance superior to camera and radar alone on the nuScenes dataset. Our source code is available at https://github.com/longyunf/rc-pda.