VIRES: Video Instance Repainting via Sketch and Text Guided Generation
This addresses the challenge of consistent and accurate video editing for users in creative and media industries, representing an incremental improvement over prior methods.
The paper tackles the problem of video instance repainting with sketch and text guidance, where existing methods struggle with temporal consistency and sketch alignment, and demonstrates that VIRES outperforms state-of-the-art methods in visual quality, temporal consistency, condition alignment, and human ratings.
We introduce VIRES, a video instance repainting method with sketch and text guidance, enabling video instance repainting, replacement, generation, and removal. Existing approaches struggle with temporal consistency and accurate alignment with the provided sketch sequence. VIRES leverages the generative priors of text-to-video models to maintain temporal consistency and produce visually pleasing results. We propose the Sequential ControlNet with the standardized self-scaling, which effectively extracts structure layouts and adaptively captures high-contrast sketch details. We further augment the diffusion transformer backbone with the sketch attention to interpret and inject fine-grained sketch semantics. A sketch-aware encoder ensures that repainted results are aligned with the provided sketch sequence. Additionally, we contribute the VireSet, a dataset with detailed annotations tailored for training and evaluating video instance editing methods. Experimental results demonstrate the effectiveness of VIRES, which outperforms state-of-the-art methods in visual quality, temporal consistency, condition alignment, and human ratings. Project page: https://hjzheng.net/projects/VIRES/