InstructVid2Vid: Controllable Video Editing with Natural Language Instructions
This addresses the problem of controllable video editing for users needing efficient, instruction-based manipulation, though it is incremental as it builds on existing models like Stable Diffusion.
The paper tackles video editing guided by natural language instructions by introducing InstructVid2Vid, a diffusion-based method that generates high-quality, temporally coherent videos for tasks like attribute editing and style transfer without per-example fine-tuning.
We introduce InstructVid2Vid, an end-to-end diffusion-based methodology for video editing guided by human language instructions. Our approach empowers video manipulation guided by natural language directives, eliminating the need for per-example fine-tuning or inversion. The proposed InstructVid2Vid model modifies a pretrained image generation model, Stable Diffusion, to generate a time-dependent sequence of video frames. By harnessing the collective intelligence of disparate models, we engineer a training dataset rich in video-instruction triplets, which is a more cost-efficient alternative to collecting data in real-world scenarios. To enhance the coherence between successive frames within the generated videos, we propose the Inter-Frames Consistency Loss and incorporate it during the training process. With multimodal classifier-free guidance during the inference stage, the generated videos is able to resonate with both the input video and the accompanying instructions. Experimental results demonstrate that InstructVid2Vid is capable of generating high-quality, temporally coherent videos and performing diverse edits, including attribute editing, background changes, and style transfer. These results underscore the versatility and effectiveness of our proposed method.