StableV2V: Stablizing Shape Consistency in Video-to-Video Editing
This addresses the issue of inferior consistency in video editing for content creators, though it appears incremental as it builds on existing motion transfer approaches.
The paper tackles the problem of shape inconsistency in video-to-video editing by introducing StableV2V, a method that decomposes editing into sequential procedures to align motions with user prompts, resulting in outperforming performance, visual consistency, and inference efficiency compared to state-of-the-art methods.
Recent advancements of generative AI have significantly promoted content creation and editing, where prevailing studies further extend this exciting progress to video editing. In doing so, these studies mainly transfer the inherent motion patterns from the source videos to the edited ones, where results with inferior consistency to user prompts are often observed, due to the lack of particular alignments between the delivered motions and edited contents. To address this limitation, we present a shape-consistent video editing method, namely StableV2V, in this paper. Our method decomposes the entire editing pipeline into several sequential procedures, where it edits the first video frame, then establishes an alignment between the delivered motions and user prompts, and eventually propagates the edited contents to all other frames based on such alignment. Furthermore, we curate a testing benchmark, namely DAVIS-Edit, for a comprehensive evaluation of video editing, considering various types of prompts and difficulties. Experimental results and analyses illustrate the outperforming performance, visual consistency, and inference efficiency of our method compared to existing state-of-the-art studies.