VLM3: Vision Language Models Are Native 3D Learners
This work provides a simplified and scalable approach for general-purpose Vision Language Models to perform 3D understanding tasks, potentially benefiting researchers and practitioners working on unified vision models.
This paper argues that Vision Language Models (VLMs) are inherently capable of 3D learning. By focusing on focal length unification, text-based pixel reference, and data mixture/scaling, the proposed VLM3 method significantly improves VLM depth estimation accuracy from 0.84 to 0.9 and achieves expert-level performance in diverse 3D tasks like pixel correspondence and camera pose estimation.
Vision Language Models (VLMs) enable a unified model to solve various vision tasks through prompting. They have shown promising performance in semantic understanding. However, 3D understanding still largely relies on expert vision models with complex task-specific designs. The key argument this work wants to make is that VLMs are native 3D learners. Our in-depth large scale study shows that 1) focal length unification, 2) text-based pixel reference and 3) data mixture and scaling, are all you need for effective 3D learning. Model architecture changes, large models, heavy data augmentations, and complex losses including the regression formulation, many of which form the foundation of expert vision models, are actually not necessary conditions. As a result, we propose VLM3, a scalable method with the simplest design that enables standard VLMs to master diverse 3D tasks. VLM3 not only advances the VLM depth estimation accuracy by a large margin (0.84 -> 0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.