VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Models
This work addresses the need for efficient and versatile video editing tools for users in multimedia and AI applications, representing a novel approach rather than an incremental improvement.
The authors tackled the problem of video editing and enhancement by proposing VIDiff, a unified foundation model that performs various video tasks based on user instructions, achieving results within seconds and demonstrating consistency for long videos through an iterative auto-regressive method.
Diffusion models have achieved significant success in image and video generation. This motivates a growing interest in video editing tasks, where videos are edited according to provided text descriptions. However, most existing approaches only focus on video editing for short clips and rely on time-consuming tuning or inference. We are the first to propose Video Instruction Diffusion (VIDiff), a unified foundation model designed for a wide range of video tasks. These tasks encompass both understanding tasks (such as language-guided video object segmentation) and generative tasks (video editing and enhancement). Our model can edit and translate the desired results within seconds based on user instructions. Moreover, we design an iterative auto-regressive method to ensure consistency in editing and enhancing long videos. We provide convincing generative results for diverse input videos and written instructions, both qualitatively and quantitatively. More examples can be found at our website https://ChenHsing.github.io/VIDiff.