Grounding Beyond Detection: Enhancing Contextual Understanding in Embodied 3D Grounding
This addresses a fundamental bottleneck in embodied AI for robotics and VR by enhancing contextual understanding in 3D object localization from human instructions.
The paper tackles the problem of embodied 3D grounding, where detection models without instruction-specific training outperform grounding models, indicating poor category-level grounding. The proposed DEGround method improves context-aware understanding and achieves a 7.52% accuracy gain over the state-of-the-art on the EmbodiedScan validation set.
Embodied 3D grounding aims to localize target objects described in human instructions from ego-centric viewpoint. Most methods typically follow a two-stage paradigm where a trained 3D detector's optimized backbone parameters are used to initialize a grounding model. In this study, we explore a fundamental question: Does embodied 3D grounding benefit enough from detection? To answer this question, we assess the grounding performance of detection models using predicted boxes filtered by the target category. Surprisingly, these detection models without any instruction-specific training outperform the grounding models explicitly trained with language instructions. This indicates that even category-level embodied 3D grounding may not be well resolved, let alone more fine-grained context-aware grounding. Motivated by this finding, we propose DEGround, which shares DETR queries as object representation for both DEtection and Grounding and enables the grounding to benefit from basic category classification and box detection. Based on this framework, we further introduce a regional activation grounding module that highlights instruction-related regions and a query-wise modulation module that incorporates sentence-level semantic into the query representation, strengthening the context-aware understanding of language instructions. Remarkably, DEGround outperforms state-of-the-art model BIP3D by 7.52% at overall accuracy on the EmbodiedScan validation set. The source code will be publicly available at https://github.com/zyn213/DEGround.