Temporal-Consistent Video Restoration with Pre-trained Diffusion Models
This work addresses video restoration for applications like virtual reality, offering an incremental improvement by enhancing temporal consistency in existing methods.
The paper tackled the problem of temporal inconsistency and approximation errors in video restoration using pre-trained diffusion models, achieving superior visual quality and temporal consistency compared to state-of-the-art methods in experiments.
Video restoration (VR) aims to recover high-quality videos from degraded ones. Although recent zero-shot VR methods using pre-trained diffusion models (DMs) show good promise, they suffer from approximation errors during reverse diffusion and insufficient temporal consistency. Moreover, dealing with 3D video data, VR is inherently computationally intensive. In this paper, we advocate viewing the reverse process in DMs as a function and present a novel Maximum a Posterior (MAP) framework that directly parameterizes video frames in the seed space of DMs, eliminating approximation errors. We also introduce strategies to promote bilevel temporal consistency: semantic consistency by leveraging clustering structures in the seed space, and pixel-level consistency by progressive warping with optical flow refinements. Extensive experiments on multiple virtual reality tasks demonstrate superior visual quality and temporal consistency achieved by our method compared to the state-of-the-art.