TempoControl: Temporal Attention Guidance for Text-to-Video Models
This addresses the lack of temporal control in generative video models for users needing specific timing in video creation, representing a novel method for a known bottleneck.
The paper tackles the problem of fine-grained temporal control in text-to-video models, introducing TempoControl to align visual concepts with timing during inference without retraining, enabling precise control over when elements appear in generated videos.
Recent advances in generative video models have enabled the creation of high-quality videos based on natural language prompts. However, these models frequently lack fine-grained temporal control, meaning they do not allow users to specify when particular visual elements should appear within a generated sequence. In this work, we introduce TempoControl, a method that allows for temporal alignment of visual concepts during inference, without requiring retraining or additional supervision. TempoControl utilizes cross-attention maps, a key component of text-to-video diffusion models, to guide the timing of concepts through a novel optimization approach. Our method steers attention using three complementary principles: aligning its temporal shape with a control signal (via correlation), amplifying it where visibility is needed (via energy), and maintaining spatial focus (via entropy). TempoControl allows precise control over timing while ensuring high video quality and diversity. We demonstrate its effectiveness across various video generation applications, including temporal reordering for single and multiple objects, as well as action and audio-aligned generation.