Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs
For MLLM-based 3D perception, this work addresses a fundamental ambiguity that limits robustness to camera changes, offering a principled solution.
The paper tackles camera intrinsic ambiguity in 3D localization for MLLMs, proposing an equation-anchored tool-use framework that explicitly uses the pinhole back-projection equation in CoT. The method outperforms baselines on 3D detection and grounding under varying camera intrinsics (0.5× to 1.5×), with largest gains at extreme scales.
3D localization in Multimodal Large Language Models (MLLMs), including 3D object detection and 3D visual grounding, is fundamentally limited by camera intrinsic ambiguity: the same image admits different 3D scenes under different cameras. Existing MLLMs either ignore camera parameters and overfit to a canonical training intrinsic, or retrieve depth and 3D cues from external tools but treat the returned values as reference cues (numerical hints that the model is free to interpret implicitly), both preventing camera information from being deterministically propagated into the prediction. We propose an equation-anchored tool-use framework that re-purposes spatial tools as formula variables. The proposed framework proactively retrieves camera intrinsics and samples multi-point metric depths, writes the pinhole back-projection equation $\hat{X} = (u_c - c_x)\bar{Z}/f_x$ explicitly in Chain-of-Thought (CoT), and substitutes tool outputs into the formula before regressing the final 9-DoF bounding box. On both 3D object detection and 3D visual grounding tasks under rescaled camera intrinsics from $0.5\times$ to $1.5\times$, our method outperforms RGB-only and tool-augmented baselines, with significant gains where the camera deviates most from the training scale. Code and data will be released.