Structure and Content-Guided Video Synthesis with Diffusion Models
This work addresses video editing for creators by enabling structure-preserving edits without expensive per-input retraining, though it is incremental as it builds on existing diffusion models.
The paper tackles the problem of editing existing videos based on textual or visual descriptions while preserving structure, by introducing a diffusion model that uses monocular depth estimates for disentanglement and joint training on images and videos to control temporal consistency, resulting in fine-grained control, customization, and strong user preference.
Text-guided generative diffusion models unlock powerful image creation and editing tools. While these have been extended to video generation, current approaches that edit the content of existing footage while retaining structure require expensive re-training for every input or rely on error-prone propagation of image edits across frames. In this work, we present a structure and content-guided video diffusion model that edits videos based on visual or textual descriptions of the desired output. Conflicts between user-provided content edits and structure representations occur due to insufficient disentanglement between the two aspects. As a solution, we show that training on monocular depth estimates with varying levels of detail provides control over structure and content fidelity. Our model is trained jointly on images and videos which also exposes explicit control of temporal consistency through a novel guidance method. Our experiments demonstrate a wide variety of successes; fine-grained control over output characteristics, customization based on a few reference images, and a strong user preference towards results by our model.