OpenScan: A Benchmark for Generalized Open-Vocabulary 3D Scene Understanding
This addresses a gap in holistic evaluation for 3D scene understanding researchers, though it is incremental as it builds on existing open-vocabulary tasks.
The paper tackles the problem of limited evaluation in open-vocabulary 3D scene understanding by introducing a more challenging task called Generalized Open-Vocabulary 3D Scene Understanding (GOV-3D) and a new benchmark named OpenScan, which includes 3D object attributes across eight linguistic aspects, and finds that state-of-the-art methods struggle with abstract vocabularies, showing a challenge not resolved by scaling object classes.
Open-vocabulary 3D scene understanding (OV-3D) aims to localize and classify novel objects beyond the closed set of object classes. However, existing approaches and benchmarks primarily focus on the open vocabulary problem within the context of object classes, which is insufficient in providing a holistic evaluation to what extent a model understands the 3D scene. In this paper, we introduce a more challenging task called Generalized Open-Vocabulary 3D Scene Understanding (GOV-3D) to explore the open vocabulary problem beyond object classes. It encompasses an open and diverse set of generalized knowledge, expressed as linguistic queries of fine-grained and object-specific attributes. To this end, we contribute a new benchmark named \textit{OpenScan}, which consists of 3D object attributes across eight representative linguistic aspects, including affordance, property, and material. We further evaluate state-of-the-art OV-3D methods on our OpenScan benchmark and discover that these methods struggle to comprehend the abstract vocabularies of the GOV-3D task, a challenge that cannot be addressed simply by scaling up object classes during training. We highlight the limitations of existing methodologies and explore promising directions to overcome the identified shortcomings.