Ges3ViG: Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understanding
This addresses a gap in embodied AI for more natural human-robot interaction by incorporating pointing gestures, though it appears incremental as it builds on existing 3D grounding methods.
The paper tackles the problem of 3D Embodied Reference Understanding (3D-ERU), which combines language descriptions and pointing gestures to identify target objects in 3D scenes, by introducing a data augmentation framework (Imputer) to create a new benchmark dataset (ImputeRefer) and proposing a novel model (Ges3ViG) that achieves ~30% accuracy improvement over other 3D-ERU models and ~9% over purely language-based models.
3-Dimensional Embodied Reference Understanding (3D-ERU) combines a language description and an accompanying pointing gesture to identify the most relevant target object in a 3D scene. Although prior work has explored pure language-based 3D grounding, there has been limited exploration of 3D-ERU, which also incorporates human pointing gestures. To address this gap, we introduce a data augmentation framework-Imputer, and use it to curate a new benchmark dataset-ImputeRefer for 3D-ERU, by incorporating human pointing gestures into existing 3D scene datasets that only contain language instructions. We also propose Ges3ViG, a novel model for 3D-ERU that achieves ~30% improvement in accuracy as compared to other 3D-ERU models and ~9% compared to other purely language-based 3D grounding models. Our code and dataset are available at https://github.com/AtharvMane/Ges3ViG.