Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initialization
This addresses the challenge of accurate text-to-video generation for users needing precise control over video content, but it is incremental as it builds on existing guidance methods without requiring model modifications.
The paper tackles the problem of text-to-video models struggling to follow text descriptions accurately, especially for spatial layouts and object trajectories, by introducing Video-MSG, a training-free guidance method that improves text alignment without fine-tuning or extra memory during inference, achieving enhanced performance on benchmarks like T2VCompBench and VBench with models such as VideoCrafter2 and CogVideoX-5B.
Recent advancements in text-to-video (T2V) diffusion models have significantly enhanced the visual quality of the generated videos. However, even recent T2V models find it challenging to follow text descriptions accurately, especially when the prompt requires accurate control of spatial layouts or object trajectories. A recent line of research uses layout guidance for T2V models that require fine-tuning or iterative manipulation of the attention map during inference time. This significantly increases the memory requirement, making it difficult to adopt a large T2V model as a backbone. To address this, we introduce Video-MSG, a training-free Guidance method for T2V generation based on Multimodal planning and Structured noise initialization. Video-MSG consists of three steps, where in the first two steps, Video-MSG creates Video Sketch, a fine-grained spatio-temporal plan for the final video, specifying background, foreground, and object trajectories, in the form of draft video frames. In the last step, Video-MSG guides a downstream T2V diffusion model with Video Sketch through noise inversion and denoising. Notably, Video-MSG does not need fine-tuning or attention manipulation with additional memory during inference time, making it easier to adopt large T2V models. Video-MSG demonstrates its effectiveness in enhancing text alignment with multiple T2V backbones (VideoCrafter2 and CogVideoX-5B) on popular T2V generation benchmarks (T2VCompBench and VBench). We provide comprehensive ablation studies about noise inversion ratio, different background generators, background object detection, and foreground object segmentation.