Multimodal Point Cloud Semantic Segmentation With Virtual Point Enhancement
This work addresses a domain-specific problem in autonomous driving and robotics by enhancing point cloud segmentation, though it is incremental as it builds on existing multimodal approaches.
The paper tackles the challenge of sparse and varying density in LiDAR-based 3D point cloud semantic segmentation, particularly for medium-range and small targets, by proposing a multimodal method with Virtual Point Enhancement, resulting in a 2.89% mIoU improvement on nuScenes with 7.7% virtual points.
LiDAR-based 3D point cloud recognition has been proven beneficial in various applications. However, the sparsity and varying density pose a significant challenge in capturing intricate details of objects, particularly for medium-range and small targets. Therefore, we propose a multi-modal point cloud semantic segmentation method based on Virtual Point Enhancement (VPE), which integrates virtual points generated from images to address these issues. These virtual points are dense but noisy, and directly incorporating them can increase computational burden and degrade performance. Therefore, we introduce a spatial difference-driven adaptive filtering module that selectively extracts valuable pseudo points from these virtual points based on density and distance, enhancing the density of medium-range targets. Subsequently, we propose a noise-robust sparse feature encoder that incorporates noise-robust feature extraction and fine-grained feature enhancement. Noise-robust feature extraction exploits the 2D image space to reduce the impact of noisy points, while fine-grained feature enhancement boosts sparse geometric features through inner-voxel neighborhood point aggregation and downsampled voxel aggregation. The results on the SemanticKITTI and nuScenes, two large-scale benchmark data sets, have validated effectiveness, significantly improving 2.89\% mIoU with the introduction of 7.7\% virtual points on nuScenes.