DiffuEraser: A Diffusion Model for Video Inpainting
This work addresses video inpainting, a problem for video editing and restoration applications, by introducing a diffusion-based approach that improves over prior methods, though it appears incremental as it builds on stable diffusion with specific enhancements.
The authors tackled video inpainting for large masks, where existing methods suffer from blurring and inconsistencies, by proposing DiffuEraser, a diffusion-based model that incorporates prior information and expanded temporal receptive fields to enhance details and coherence. The method outperforms state-of-the-art techniques in content completeness and temporal consistency.
Recent video inpainting algorithms integrate flow-based pixel propagation with transformer-based generation to leverage optical flow for restoring textures and objects using information from neighboring frames, while completing masked regions through visual Transformers. However, these approaches often encounter blurring and temporal inconsistencies when dealing with large masks, highlighting the need for models with enhanced generative capabilities. Recently, diffusion models have emerged as a prominent technique in image and video generation due to their impressive performance. In this paper, we introduce DiffuEraser, a video inpainting model based on stable diffusion, designed to fill masked regions with greater details and more coherent structures. We incorporate prior information to provide initialization and weak conditioning,which helps mitigate noisy artifacts and suppress hallucinations. Additionally, to improve temporal consistency during long-sequence inference, we expand the temporal receptive fields of both the prior model and DiffuEraser, and further enhance consistency by leveraging the temporal smoothing property of Video Diffusion Models. Experimental results demonstrate that our proposed method outperforms state-of-the-art techniques in both content completeness and temporal consistency while maintaining acceptable efficiency.