DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models
This provides a versatile zero-shot solution for video enhancement, addressing temporal inconsistencies without task-specific training, which is incremental as it builds on existing image models.
The paper tackles the problem of applying image restoration diffusion models to video without retraining, achieving high-quality video restoration with superior temporal consistency across diverse datasets and degradation conditions, including 8x super-resolution and severe noise.
We present DiffIR2VR-Zero, a zero-shot framework that enables any pre-trained image restoration diffusion model to perform high-quality video restoration without additional training. While image diffusion models have shown remarkable restoration capabilities, their direct application to video leads to temporal inconsistencies, and existing video restoration methods require extensive retraining for different degradation types. Our approach addresses these challenges through two key innovations: a hierarchical latent warping strategy that maintains consistency across both keyframes and local frames, and a hybrid token merging mechanism that adaptively combines optical flow and feature matching. Through extensive experiments, we demonstrate that our method not only maintains the high-quality restoration of base diffusion models but also achieves superior temporal consistency across diverse datasets and degradation conditions, including challenging scenarios like 8$\times$ super-resolution and severe noise. Importantly, our framework works with any image restoration diffusion model, providing a versatile solution for video enhancement without task-specific training or modifications. Project page: https://jimmycv07.github.io/DiffIR2VR_web/