Learning Point-Language Hierarchical Alignment for 3D Visual Grounding
It addresses the problem of aligning language with 3D scenes for tasks like robotics and AR/VR, representing an incremental improvement with novel mechanisms.
The paper tackles 3D visual grounding by proposing a hierarchical alignment model (HAM) that learns multi-granularity visual and linguistic representations, achieving state-of-the-art performance and winning the ECCV 2022 ScanRefer challenge.
This paper presents a novel hierarchical alignment model (HAM) that learns multi-granularity visual and linguistic representations in an end-to-end manner. We extract key points and proposal points to model 3D contexts and instances, and propose point-language alignment with context modulation (PLACM) mechanism, which learns to gradually align word-level and sentence-level linguistic embeddings with visual representations, while the modulation with the visual context captures latent informative relationships. To further capture both global and local relationships, we propose a spatially multi-granular modeling scheme that applies PLACM to both global and local fields. Experimental results demonstrate the superiority of HAM, with visualized results showing that it can dynamically model fine-grained visual and linguistic representations. HAM outperforms existing methods by a significant margin and achieves state-of-the-art performance on two publicly available datasets, and won the championship in ECCV 2022 ScanRefer challenge. Code is available at~\url{https://github.com/PPjmchen/HAM}.