Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusion
This work addresses the challenge of 3D video generation for computer vision applications, suggesting a simpler approach that could reduce complexity in generative algorithms.
The paper tackles the problem of generating dynamic 3D scenes by investigating whether explicit modeling of 3D temporal dynamics is necessary, proposing Vid3D which uses 2D video diffusion to generate 3D videos independently per timestep, and finds it achieves comparable results to state-of-the-art methods without such modeling.
A recent frontier in computer vision has been the task of 3D video generation, which consists of generating a time-varying 3D representation of a scene. To generate dynamic 3D scenes, current methods explicitly model 3D temporal dynamics by jointly optimizing for consistency across both time and views of the scene. In this paper, we instead investigate whether it is necessary to explicitly enforce multiview consistency over time, as current approaches do, or if it is sufficient for a model to generate 3D representations of each timestep independently. We hence propose a model, Vid3D, that leverages 2D video diffusion to generate 3D videos by first generating a 2D "seed" of the video's temporal dynamics and then independently generating a 3D representation for each timestep in the seed video. We evaluate Vid3D against two state-of-the-art 3D video generation methods and find that Vid3D is achieves comparable results despite not explicitly modeling 3D temporal dynamics. We further ablate how the quality of Vid3D depends on the number of views generated per frame. While we observe some degradation with fewer views, performance degradation remains minor. Our results thus suggest that 3D temporal knowledge may not be necessary to generate high-quality dynamic 3D scenes, potentially enabling simpler generative algorithms for this task.