CoordRefer: Coordinate-Aware 3D Visual Grounding from Multiview Images
This work improves 3D visual grounding for embodied AI and robotics by resolving a key ambiguity in coordinate-based prediction, offering a practical solution for multiview image inputs.
CoordRefer addresses the ambiguity in multiview image-based 3D visual grounding by decoupling coordinate frame selection from coordinate-conditioned box regression. It achieves 11% improvement in Acc@0.25 and 7% in Acc@0.5 over a coordinate-agnostic baseline on ScanRefer with Qwen3-VL-2B, and its refined variant outperforms methods using explicit 3D inputs.
Multiview image-based 3D visual grounding predicts a coordinate frame to define a coordinate system and then regresses a 3D bounding box for localization. However, existing methods jointly optimize coordinate frame selection and box regression, leading to coordinate-relative box ambiguity and degraded grounding performance. This ambiguity arises because the same box admits different numerical representations across coordinate frames, creating multiple optimization targets and yielding invalid compromise predictions. To tackle this challenge, we propose CoordRefer, a coordinate-aware framework that decouples coordinate frame selection from coordinate-conditioned grounding. CoordRefer first selects a reference frame to define the coordinate system and then conditions 3D box prediction on the coordinate system. We perform coordinate-aware supervised fine-tuning to establish coordinate frame selection and coordinate-conditioned box regression, followed by Group Relative Policy Optimization with 3D IoU-based rewards to align both stages with downstream grounding quality. On ScanRefer with Qwen3-VL-2B, CoordRefer achieves gains of 11% in Acc@0.25 and 7% in Acc@0.5 over the coordinate-agnostic baseline, while its geometrically refined variant surpasses methods using explicit 3D inputs.