Vanilla ViT for Automotive Point Cloud Semantic Segmentation
This work is significant for researchers and practitioners in autonomous driving and 3D computer vision, offering a simpler, unified architecture for point cloud segmentation.
This paper tackles the problem of applying vanilla Vision Transformers (ViTs) to large-scale automotive lidar scene semantic segmentation, a domain typically dominated by U-Net architectures. The authors successfully bridge the performance gap, achieving results that match or exceed state-of-the-art methods.
Plain Transformers have become the de-facto architecture for processing text, audio, image, and video, offering a unified backbone for multimodal learning. However, state-of-the-art architectures for point cloud semantic segmentation remain dominated by U-Nets architectures where convolutions are interleaved with local or windowed attentions. In this work, we show how to effectively leverage vanilla, non-hierarchical ViTs for segmentation of large-scale automotive lidar scenes. We bridge the performance gap thanks to a carefully designed tokenizer, a lightweight decoder segmentation head, and tailored data augmentations. Our approach, VaViT for Vanilla ViT, matches or exceeds the performance of state-of-the-art methods while maintaining the simplicity of ViT architecture. We provide extensive evaluations on nuScenes, SemanticKITTI, and Waymo Open Dataset to validate the efficiency of our method. Code and models are available at https://github.com/valeoai/VaViT.