Scaling Diverse Language Generation for 3D Visual Grounding
For researchers in 3D visual grounding, this work addresses the lack of diverse training data, but the improvements are incremental over existing methods.
The paper proposes ViGiL3D++, a scalable method for generating diverse 3D visual grounding queries by combining constraint sampling with LLMs, which improves model performance on several 3DVG benchmarks while revealing limitations of VLMs.
Developing robust models for 3D visual grounding (3DVG), the localization of entities in a 3D scene described in natural language, is important for enabling agents to correspond spatial language with objects in the physical world. However, the lack of diverse descriptions at scale prevents models from generalizing beyond simple linguistic patterns. Recent such attempts lack diversity in the constraint types and language used to ground objects. Captioning methods cannot precisely contrast objects, which is important for visual grounding. We therefore propose ViGiL3D++, a scalable, scene-agnostic method that generates diverse visual grounding queries by combining constraint sampling in scene graphs with the language generation of LLMs. We show that it has greater diversity over existing scaled datasets and improves model performance over several 3DVG benchmarks but also illuminates outstanding limitations of VLMs.