Language-Assisted 3D Scene Understanding
This work addresses the challenge of improving 3D scene understanding for applications like autonomous driving and robotics, though it is incremental as it builds on existing multi-modal methods.
The paper tackles the problem of limited point cloud datasets by proposing a language-assisted approach (LAST-PCL) that uses LLMs for text enrichment and training-free feature selection, achieving state-of-the-art or comparable performance in 3D semantic segmentation, object detection, and scene classification tasks.
The scale and quality of point cloud datasets constrain the advancement of point cloud learning. Recently, with the development of multi-modal learning, the incorporation of domain-agnostic prior knowledge from other modalities, such as images and text, to assist in point cloud feature learning has been considered a promising avenue. Existing methods have demonstrated the effectiveness of multi-modal contrastive training and feature distillation on point clouds. However, challenges remain, including the requirement for paired triplet data, redundancy and ambiguity in supervised features, and the disruption of the original priors. In this paper, we propose a language-assisted approach to point cloud feature learning (LAST-PCL), enriching semantic concepts through LLMs-based text enrichment. We achieve de-redundancy and feature dimensionality reduction without compromising textual priors by statistical-based and training-free significant feature selection. Furthermore, we also delve into an in-depth analysis of the impact of text contrastive training on the point cloud. Extensive experiments validate that the proposed method learns semantically meaningful point cloud features and achieves state-of-the-art or comparable performance in 3D semantic segmentation, 3D object detection, and 3D scene classification tasks.