S^2VG: 3D Stereoscopic and Spatial Video Generation via Denoising Frame Matrix
This work addresses the problem of creating immersive 3D content for applications like VR/AR, offering an incremental advancement by building on existing monocular video generation models without requiring fine-tuning.
The paper tackles the challenge of generating 3D stereoscopic and spatial videos from monocular video models by proposing a pose-free, training-free method that warps videos into multiple viewpoints and uses a frame matrix inpainting framework with a dual-update scheme, achieving significant improvements over previous methods as validated on videos from models like Sora and Lumiere.
While video generation models excel at producing high-quality monocular videos, generating 3D stereoscopic and spatial videos for immersive applications remains an underexplored challenge. We present a pose-free and training-free method that leverages an off-the-shelf monocular video generation model to produce immersive 3D videos. Our approach first warps the generated monocular video into pre-defined camera viewpoints using estimated depth information, then applies a novel \textit{frame matrix} inpainting framework. This framework utilizes the original video generation model to synthesize missing content across different viewpoints and timestamps, ensuring spatial and temporal consistency without requiring additional model fine-tuning. Moreover, we develop a \dualupdate~scheme that further improves the quality of video inpainting by alleviating the negative effects propagated from disoccluded areas in the latent space. The resulting multi-view videos are then adapted into stereoscopic pairs or optimized into 4D Gaussians for spatial video synthesis. We validate the efficacy of our proposed method by conducting experiments on videos from various generative models, such as Sora, Lumiere, WALT, and Zeroscope. The experiments demonstrate that our method has a significant improvement over previous methods. Project page at: https://daipengwa.github.io/S-2VG_ProjectPage/