Multi-View Transformer for 3D Visual Grounding
This addresses the challenge of robust 3D object localization from language for applications like robotics and AR/VR, representing a strong specific gain in a domain-specific area.
The paper tackles the problem of 3D visual grounding, where natural language descriptions are matched to objects in 3D scenes, by proposing a Multi-View Transformer to eliminate dependence on specific views, resulting in performance improvements of 11.2% and 7.1% over state-of-the-art methods on two datasets.
The 3D visual grounding task aims to ground a natural language description to the targeted object in a 3D scene, which is usually represented in 3D point clouds. Previous works studied visual grounding under specific views. The vision-language correspondence learned by this way can easily fail once the view changes. In this paper, we propose a Multi-View Transformer (MVT) for 3D visual grounding. We project the 3D scene to a multi-view space, in which the position information of the 3D scene under different views are modeled simultaneously and aggregated together. The multi-view space enables the network to learn a more robust multi-modal representation for 3D visual grounding and eliminates the dependence on specific views. Extensive experiments show that our approach significantly outperforms all state-of-the-art methods. Specifically, on Nr3D and Sr3D datasets, our method outperforms the best competitor by 11.2% and 7.1% and even surpasses recent work with extra 2D assistance by 5.9% and 6.6%. Our code is available at https://github.com/sega-hsj/MVT-3DVG.