MotionBridge: Dynamic Video Inbetweening with Flexible Controls
This work addresses video editing and synthesis challenges by providing fine control over intermediate frames, though it appears incremental as it builds on existing video generation techniques.
The paper tackles the problem of generating smooth transitions between video frames with flexible control over details, introducing MotionBridge which enables multi-modal controls like trajectory strokes and text, achieving dynamic and contextually accurate results.
By generating plausible and smooth transitions between two image frames, video inbetweening is an essential tool for video editing and long video synthesis. Traditional works lack the capability to generate complex large motions. While recent video generation techniques are powerful in creating high-quality results, they often lack fine control over the details of intermediate frames, which can lead to results that do not align with the creative mind. We introduce MotionBridge, a unified video inbetweening framework that allows flexible controls, including trajectory strokes, keyframes, masks, guide pixels, and text. However, learning such multi-modal controls in a unified framework is a challenging task. We thus design two generators to extract the control signal faithfully and encode feature through dual-branch embedders to resolve ambiguities. We further introduce a curriculum training strategy to smoothly learn various controls. Extensive qualitative and quantitative experiments have demonstrated that such multi-modal controls enable a more dynamic, customizable, and contextually accurate visual narrative.