Where, Not What: Compelling Video LLMs to Learn Geometric Causality for 3D-Grounding
This addresses a specific bottleneck in 3D spatial reasoning for VLMs, with incremental improvements in geometric causality learning.
The paper tackles the problem of 2D semantic bias in multimodal 3D grounding for Vision-Language Models, which causes suboptimal fusion performance by over-relying on 2D image features. The proposed W2R2 training framework achieves significant gains in localization accuracy and robustness on ScanRefer and ScanQA datasets, particularly in cluttered outdoor scenes.
Multimodal 3D grounding has garnered considerable interest in Vision-Language Models (VLMs) \cite{yin2025spatial} for advancing spatial reasoning in complex environments. However, these models suffer from a severe "2D semantic bias" that arises from over-reliance on 2D image features for coarse localization, largely disregarding 3D geometric inputs and resulting in suboptimal fusion performance. In this paper, we propose a novel training framework called What-Where Representation Re-Forming (W2R2) to tackle this issue via disentangled representation learning and targeted shortcut suppression. Our approach fundamentally reshapes the model's internal space by designating 2D features as semantic beacons for "What" identification and 3D features as spatial anchors for "Where" localization, enabling precise 3D grounding without modifying inference architecture. Key components include a dual-objective loss function with an Alignment Loss that supervises fused predictions using adapted cross-entropy for multimodal synergy, and a Pseudo-Label Loss that penalizes overly effective 2D-dominant pseudo-outputs via a margin-based mechanism. Experiments conducted on ScanRefer and ScanQA demonstrate the effectiveness of W2R2, with significant gains in localization accuracy and robustness, particularly in cluttered outdoor scenes.