Elastic3D: Controllable Stereo Video Conversion with Guided Latent Decoding
This addresses the growing demand for immersive 3D content by providing a more reliable and controllable solution for video conversion, though it appears incremental as it builds on latent diffusion with a novel decoder component.
The paper tackles the problem of automated monocular-to-stereo video conversion by introducing Elastic3D, a controllable end-to-end method based on latent diffusion that avoids artifacts from explicit depth estimation and warping. The result is a method that outperforms existing baselines on three real-world datasets and allows user control over the stereo effect strength.
The growing demand for immersive 3D content calls for automated monocular-to-stereo video conversion. We present Elastic3D, a controllable, direct end-to-end method for upgrading a conventional video to a binocular one. Our approach, based on (conditional) latent diffusion, avoids artifacts due to explicit depth estimation and warping. The key to its high-quality stereo video output is a novel, guided VAE decoder that ensures sharp and epipolar-consistent stereo video output. Moreover, our method gives the user control over the strength of the stereo effect (more precisely, the disparity range) at inference time, via an intuitive, scalar tuning knob. Experiments on three different datasets of real-world stereo videos show that our method outperforms both traditional warping-based and recent warping-free baselines and sets a new standard for reliable, controllable stereo video conversion. Please check the project page for the video samples https://elastic3d.github.io.