MotionCrafter: Dense Geometry and Motion Reconstruction with a 4D VAE
This work addresses the challenge of accurate 4D scene understanding from single-camera videos, which is crucial for applications like robotics and augmented reality, and represents a significant advance over prior methods.
MotionCrafter tackles the problem of jointly reconstructing 4D geometry and estimating dense motion from monocular video, achieving state-of-the-art performance with 38.64% and 25.0% improvements in geometry and motion reconstruction, respectively, without post-optimization.
We introduce MotionCrafter, a video diffusion-based framework that jointly reconstructs 4D geometry and estimates dense motion from a monocular video. The core of our method is a novel joint representation of dense 3D point maps and 3D scene flows in a shared coordinate system, and a novel 4D VAE to effectively learn this representation. Unlike prior work that forces the 3D value and latents to align strictly with RGB VAE latents-despite their fundamentally different distributions-we show that such alignment is unnecessary and leads to suboptimal performance. Instead, we introduce a new data normalization and VAE training strategy that better transfers diffusion priors and greatly improves reconstruction quality. Extensive experiments across multiple datasets demonstrate that MotionCrafter achieves state-of-the-art performance in both geometry reconstruction and dense scene flow estimation, delivering 38.64% and 25.0% improvements in geometry and motion reconstruction, respectively, all without any post-optimization. Project page: https://ruijiezhu94.github.io/MotionCrafter_Page