Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos
This addresses the challenge of obtaining ground truth annotations for 3D motion recovery, enabling applications in robotics and scene reconstruction.
The authors tackled the problem of learning dynamic 3D scene understanding from imagery by mining high-quality 4D reconstructions from internet stereo videos, generating large-scale pseudo-metric 3D point clouds with motion trajectories, and training a model that generalizes to diverse real-world scenes.
Learning to understand dynamic 3D scenes from imagery is crucial for applications ranging from robotics to scene reconstruction. Yet, unlike other problems where large-scale supervised training has enabled rapid progress, directly supervising methods for recovering 3D motion remains challenging due to the fundamental difficulty of obtaining ground truth annotations. We present a system for mining high-quality 4D reconstructions from internet stereoscopic, wide-angle videos. Our system fuses and filters the outputs of camera pose estimation, stereo depth estimation, and temporal tracking methods into high-quality dynamic 3D reconstructions. We use this method to generate large-scale data in the form of world-consistent, pseudo-metric 3D point clouds with long-term motion trajectories. We demonstrate the utility of this data by training a variant of DUSt3R to predict structure and 3D motion from real-world image pairs, showing that training on our reconstructed data enables generalization to diverse real-world scenes. Project page and data at: https://stereo4d.github.io