Multimodal Transformer for Automatic 3D Annotation and Object Detection
This addresses the high human effort in creating 3D datasets for autonomous driving, offering an incremental improvement over existing autolabelers.
The paper tackles the problem of automating 3D box annotation for LiDAR scans by proposing a multimodal transformer that uses images to densify sparse point clouds and generate precise 3D annotations from weak 2D boxes, achieving 4.48% and 4.03% better 3D AP on KITTI moderate and hard samples compared to the state-of-the-art.
Despite a growing number of datasets being collected for training 3D object detection models, significant human effort is still required to annotate 3D boxes on LiDAR scans. To automate the annotation and facilitate the production of various customized datasets, we propose an end-to-end multimodal transformer (MTrans) autolabeler, which leverages both LiDAR scans and images to generate precise 3D box annotations from weak 2D bounding boxes. To alleviate the pervasive sparsity problem that hinders existing autolabelers, MTrans densifies the sparse point clouds by generating new 3D points based on 2D image information. With a multi-task design, MTrans segments the foreground/background, densifies LiDAR point clouds, and regresses 3D boxes simultaneously. Experimental results verify the effectiveness of the MTrans for improving the quality of the generated labels. By enriching the sparse point clouds, our method achieves 4.48\% and 4.03\% better 3D AP on KITTI moderate and hard samples, respectively, versus the state-of-the-art autolabeler. MTrans can also be extended to improve the accuracy for 3D object detection, resulting in a remarkable 89.45\% AP on KITTI hard samples. Codes are at \url{https://github.com/Cliu2/MTrans}.