OnlyFlow: Optical Flow based Motion Conditioning for Video Diffusion Models
This addresses the need for versatile motion control in video generation applications like camera movement and editing, though it is incremental as it builds on existing text-to-video models.
The paper tackles the problem of text-to-video generation with precise motion control by proposing OnlyFlow, which uses optical flow from an input video to condition generated videos, resulting in performance that positively compares to state-of-the-art methods on a wide range of tasks.
We consider the problem of text-to-video generation tasks with precise control for various applications such as camera movement control and video-to-video editing. Most methods tacking this problem rely on providing user-defined controls, such as binary masks or camera movement embeddings. In our approach we propose OnlyFlow, an approach leveraging the optical flow firstly extracted from an input video to condition the motion of generated videos. Using a text prompt and an input video, OnlyFlow allows the user to generate videos that respect the motion of the input video as well as the text prompt. This is implemented through an optical flow estimation model applied on the input video, which is then fed to a trainable optical flow encoder. The output feature maps are then injected into the text-to-video backbone model. We perform quantitative, qualitative and user preference studies to show that OnlyFlow positively compares to state-of-the-art methods on a wide range of tasks, even though OnlyFlow was not specifically trained for such tasks. OnlyFlow thus constitutes a versatile, lightweight yet efficient method for controlling motion in text-to-video generation. Models and code will be made available on GitHub and HuggingFace.