Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting
This addresses the data bottleneck for researchers and developers in spatial AI, enabling scalable 3D data generation from abundant 2D imagery, though it is incremental as it builds on existing depth estimation and calibration techniques.
The paper tackles the scarcity of large-scale 3D datasets by developing a pipeline that converts single-view images into realistic 3D representations, generating datasets like COCO-3D and Objects365-v2-3D to benefit various 3D tasks.
Spatial intelligence is emerging as a transformative frontier in AI, yet it remains constrained by the scarcity of large-scale 3D datasets. Unlike the abundant 2D imagery, acquiring 3D data typically requires specialized sensors and laborious annotation. In this work, we present a scalable pipeline that converts single-view images into comprehensive, scale- and appearance-realistic 3D representations - including point clouds, camera poses, depth maps, and pseudo-RGBD - via integrated depth estimation, camera calibration, and scale calibration. Our method bridges the gap between the vast repository of imagery and the increasing demand for spatial scene understanding. By automatically generating authentic, scale-aware 3D data from images, we significantly reduce data collection costs and open new avenues for advancing spatial intelligence. We release two generated spatial datasets, i.e., COCO-3D and Objects365-v2-3D, and demonstrate through extensive experiments that our generated data can benefit various 3D tasks, ranging from fundamental perception to MLLM-based reasoning. These results validate our pipeline as an effective solution for developing AI systems capable of perceiving, understanding, and interacting with physical environments.