ShapeSplat: A Large-scale Dataset of Gaussian Splats and Their Self-Supervised Pretraining
This provides a dataset and pretraining method for 3DGS representation learning, which is incremental but addresses a specific bottleneck in 3D vision research.
The authors tackled the lack of large-scale datasets for 3D Gaussian Splatting (3DGS) by creating ShapeSplat, a dataset of 206K objects across 87 categories using 3.8 GPU years of computation, and they introduced Gaussian-MAE for unsupervised pretraining, showing that optimized GS centroids degrade classification but improve segmentation, with their grouping method leading to notable gains in finetuning tasks.
3D Gaussian Splatting (3DGS) has become the de facto method of 3D representation in many vision tasks. This calls for the 3D understanding directly in this representation space. To facilitate the research in this direction, we first build ShapeSplat, a large-scale dataset of 3DGS using the commonly used ShapeNet, ModelNet and Objaverse datasets. Our dataset ShapeSplat consists of 206K objects spanning over 87 unique categories, whose labels are in accordance with the respective datasets. The creation of this dataset utilized the compute equivalent of 3.8 GPU years on a TITAN XP GPU. We utilize our dataset for unsupervised pretraining and supervised finetuning for classification and segmentation tasks. To this end, we introduce Gaussian-MAE, which highlights the unique benefits of representation learning from Gaussian parameters. Through exhaustive experiments, we provide several valuable insights. In particular, we show that (1) the distribution of the optimized GS centroids significantly differs from the uniformly sampled point cloud (used for initialization) counterpart; (2) this change in distribution results in degradation in classification but improvement in segmentation tasks when using only the centroids; (3) to leverage additional Gaussian parameters, we propose Gaussian feature grouping in a normalized feature space, along with splats pooling layer, offering a tailored solution to effectively group and embed similar Gaussians, which leads to notable improvement in finetuning tasks.