Mono3DVG: 3D Visual Grounding in Monocular Images
This addresses the challenge of localizing 3D objects from monocular images using text for applications in robotics or autonomous systems, but it is incremental as it builds on existing visual grounding and 3D perception tasks.
The paper tackles the problem of 3D visual grounding in monocular RGB images using language descriptions with appearance and geometry information, and the result is a method that significantly outperforms all baselines, as shown through extensive comparisons and ablation studies.
We introduce a novel task of 3D visual grounding in monocular RGB images using language descriptions with both appearance and geometry information. Specifically, we build a large-scale dataset, Mono3DRefer, which contains 3D object targets with their corresponding geometric text descriptions, generated by ChatGPT and refined manually. To foster this task, we propose Mono3DVG-TR, an end-to-end transformer-based network, which takes advantage of both the appearance and geometry information in text embeddings for multi-modal learning and 3D object localization. Depth predictor is designed to explicitly learn geometry features. The dual text-guided adapter is proposed to refine multiscale visual and geometry features of the referred object. Based on depth-text-visual stacking attention, the decoder fuses object-level geometric cues and visual appearance into a learnable query. Comprehensive benchmarks and some insightful analyses are provided for Mono3DVG. Extensive comparisons and ablation studies show that our method significantly outperforms all baselines. The dataset and code will be publicly available at: https://github.com/ZhanYang-nwpu/Mono3DVG.