Evaluating Foundation Models' 3D Understanding Through Multi-View Correspondence Analysis
This work addresses the need for better benchmarking of 3D reasoning in foundation models for applications like robotics and autonomous driving, though it is incremental as it extends an existing 2D framework to 3D.
The authors tackled the problem of evaluating the intrinsic 3D spatial understanding of foundation models by introducing a novel benchmark that requires no fine-tuning, using the MVImgNet dataset to test segmentation across viewpoint shifts, and found that DINO-based encoders performed competitively, with scores reported in easy, medium, hard, and extreme categories.
Benchmarking 3D spatial understanding of foundation models is essential for real-world applications such as robotics and autonomous driving. Existing evaluations often rely on downstream fine-tuning with linear heads or task-specific decoders, making it difficult to isolate the intrinsic 3D reasoning ability of pre-trained encoders. In this work, we introduce a novel benchmark for in-context 3D scene understanding that requires no fine-tuning and directly probes the quality of dense visual features. Building on the Hummingbird framework, which evaluates in-context 2D scene understanding, we extend the setup to the 3D Multi-View ImageNet (MVImgNet) dataset. Given a set of images depicting objects at specific camera angles (keys), we benchmark the performance of segmenting novel views (queries) and report the scores in 4 categories of easy, medium, hard, and extreme based on the key-query view contrast. We benchmark 7 state-of-the-art foundation models and show that DINO-based encoders remain competitive across large viewpoint shifts. Our code is publicly available at https://github.com/ToyeshC/open-hummingbird-3d-eval.