In-Context Learning with Unpaired Clips for Instruction-based Video Editing
This work addresses the problem of high-cost dataset construction for video editing, enabling more efficient training for researchers and practitioners in video generation and editing.
The paper tackles the challenge of instruction-based video editing by introducing a low-cost pretraining strategy using unpaired video clips, which reduces the need for large-scale paired datasets. The method achieves a 12% improvement in instruction alignment and a 15% improvement in visual fidelity compared to existing approaches.
Despite the rapid progress of instruction-based image editing, its extension to video remains underexplored, primarily due to the prohibitive cost and complexity of constructing large-scale paired video editing datasets. To address this challenge, we introduce a low-cost pretraining strategy for instruction-based video editing that leverages in-context learning from unpaired video clips. We show that pretraining a foundation video generation model with this strategy endows it with general editing capabilities, such as adding, replacing, or deleting operations, according to input editing instructions. The pretrained model can then be efficiently refined with a small amount of high-quality paired editing data. Built upon HunyuanVideoT2V, our framework first pretrains on approximately 1M real video clips to learn basic editing concepts, and subsequently fine-tunes on fewer than 150k curated editing pairs to extend more editing tasks and improve the editing quality. Comparative experiments show that our method surpasses existing instruction-based video editing approaches in both instruction alignment and visual fidelity, achieving a 12\% improvement in editing instruction following and a 15\% improvement in editing quality.