ISETAuto: Detecting vehicles with depth and radiance information
This research provides insights for autonomous driving system designers by quantifying the benefits of combining depth and radiance data for vehicle detection, addressing limitations of individual sensor types.
This paper investigates vehicle detection using depth maps (LiDAR) and radiance images (cameras), or both, with a ResNet model. It found that depth maps outperform radiance images at typical camera resolutions, but radiance images are superior when depth map resolution drops to LiDAR levels. Combining both depth and radiance information significantly improves detection, achieving higher average precision than using either alone.
Autonomous driving applications use two types of sensor systems to identify vehicles - depth sensing LiDAR and radiance sensing cameras. We compare the performance (average precision) of a ResNet for vehicle detection in complex, daytime, driving scenes when the input is a depth map (D = d(x,y)), a radiance image (L = r(x,y)), or both [D,L]. (1) When the spatial sampling resolution of the depth map and radiance image are equal to typical camera resolutions, a ResNet detects vehicles at higher average precision from depth than radiance. (2) As the spatial sampling of the depth map declines to the range of current LiDAR devices, the ResNet average precision is higher for radiance than depth. (3) For a hybrid system that combines a depth map and radiance image, the average precision is higher than using depth or radiance alone. We established these observations in simulation and then confirmed them using realworld data. The advantage of combining depth and radiance can be explained by noting that the two type of information have complementary weaknesses. The radiance data are limited by dynamic range and motion blur. The LiDAR data have relatively low spatial resolution. The ResNet combines the two data sources effectively to improve overall vehicle detection.