Auto-Vocabulary 3D Object Detection
This addresses the limitation of existing methods that rely on user-specified classes, enabling more flexible and automated 3D scene understanding for applications like robotics and augmented reality.
The paper tackles the problem of open-vocabulary 3D object detection by proposing Auto-Vocabulary 3D Object Detection (AV3DOD), which automatically generates class names for detected objects without user input, achieving state-of-the-art performance with a 3.48 mAP improvement and 24.5% relative gain in semantic quality on ScanNetV2.
Open-vocabulary 3D object detection methods are able to localize 3D boxes of classes unseen during training. Despite the name, existing methods rely on user-specified classes both at training and inference. We propose to study Auto-Vocabulary 3D Object Detection (AV3DOD), where the classes are automatically generated for the detected objects without any user input. To this end, we introduce Semantic Score (SS) to evaluate the quality of the generated class names. We then develop a novel framework, AV3DOD, which leverages 2D vision-language models (VLMs) to generate rich semantic candidates through image captioning, pseudo 3D box generation, and feature-space semantics expansion. AV3DOD achieves the state-of-the-art (SOTA) performance on both localization (mAP) and semantic quality (SS) on the ScanNetV2 and SUNRGB-D datasets. Notably, it surpasses the SOTA, CoDA, by 3.48 overall mAP and attains a 24.5% relative improvement in SS on ScanNetV2.