M2SVid: End-to-End Inpainting and Refinement for Monocular-to-Stereo Video Conversion
This addresses the problem of generating high-quality stereo videos from monocular inputs for applications like VR and 3D media, representing an incremental improvement over existing techniques.
The paper tackles monocular-to-stereo video conversion by proposing an end-to-end architecture for inpainting and refining warped right views, outperforming previous methods with an average rank of 1.43 in a user study and being 6x faster than the second-best method.
We tackle the problem of monocular-to-stereo video conversion and propose a novel architecture for inpainting and refinement of the warped right view obtained by depth-based reprojection of the input left view. We extend the Stable Video Diffusion (SVD) model to utilize the input left video, the warped right video, and the disocclusion masks as conditioning input to generate a high-quality right camera view. In order to effectively exploit information from neighboring frames for inpainting, we modify the attention layers in SVD to compute full attention for discoccluded pixels. Our model is trained to generate the right view video in an end-to-end manner by minimizing image space losses to ensure high-quality generation. Our approach outperforms previous state-of-the-art methods, obtaining an average rank of 1.43 among the 4 compared methods in a user study, while being 6x faster than the second placed method.