Label Name is Mantra: Unifying Point Cloud Segmentation across Heterogeneous Datasets
This addresses the practical usability issue of point cloud segmentation for applications like 3D vision by enabling learning from heterogeneous datasets, though it is incremental in combining existing techniques.
The paper tackles the problem of point cloud segmentation across datasets with mismatched labels by using a pre-trained language model to embed label names into a continuous space, enabling joint training and improving generalization. It reports outperforming state-of-the-art methods by a large margin in experiments.
Point cloud segmentation is a fundamental task in 3D vision that serves a wide range of applications. Although great progresses have been made these years, its practical usability is still limited by the availability of training data. Existing approaches cannot make full use of multiple datasets on hand due to the label mismatch among different datasets. In this paper, we propose a principled approach that supports learning from heterogeneous datasets with different label sets. Our idea is to utilize a pre-trained language model to embed discrete labels to a continuous latent space with the help of their label names. This unifies all labels of different datasets, so that joint training is doable. Meanwhile, classifying points in the continuous 3D space by their vocabulary tokens significantly increase the generalization ability of the model in comparison with existing approaches that have fixed decoder architecture. Besides, we also integrate prompt learning in our framework to alleviate data shifts among different data sources. Extensive experiments demonstrate that our model outperforms the state-of-the-art by a large margin.