UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding
This work addresses the challenge of multimodal understanding in 3D vision-language tasks for applications like robotics and augmented reality, presenting an incremental improvement over previous task-specific methods.
The paper tackles the problem of jointly learning 3D dense captioning and visual grounding by proposing UniT3D, a unified transformer architecture, which achieves significant performance gains on these tasks through supervised joint pre-training.
Performing 3D dense captioning and visual grounding requires a common and shared understanding of the underlying multimodal relationships. However, despite some previous attempts on connecting these two related tasks with highly task-specific neural modules, it remains understudied how to explicitly depict their shared nature to learn them simultaneously. In this work, we propose UniT3D, a simple yet effective fully unified transformer-based architecture for jointly solving 3D visual grounding and dense captioning. UniT3D enables learning a strong multimodal representation across the two tasks through a supervised joint pre-training scheme with bidirectional and seq-to-seq objectives. With a generic architecture design, UniT3D allows expanding the pre-training scope to more various training sources such as the synthesized data from 2D prior knowledge to benefit 3D vision-language tasks. Extensive experiments and analysis demonstrate that UniT3D obtains significant gains for 3D dense captioning and visual grounding.