AADiff: Audio-Aligned Video Synthesis with Text-to-Image Diffusion
This addresses the challenge of generating more dynamic and synchronized videos for content creators, though it appears incremental as it builds on existing diffusion models.
The paper tackles the problem of text-to-video synthesis lacking detailed temporal dynamics by introducing a framework that uses audio signals to control temporal alignment, enabling an off-the-shelf text-to-image diffusion model to generate audio-aligned videos with improved temporal flexibility and coherence.
Recent advances in diffusion models have showcased promising results in the text-to-video (T2V) synthesis task. However, as these T2V models solely employ text as the guidance, they tend to struggle in modeling detailed temporal dynamics. In this paper, we introduce a novel T2V framework that additionally employ audio signals to control the temporal dynamics, empowering an off-the-shelf T2I diffusion to generate audio-aligned videos. We propose audio-based regional editing and signal smoothing to strike a good balance between the two contradicting desiderata of video synthesis, i.e., temporal flexibility and coherence. We empirically demonstrate the effectiveness of our method through experiments, and further present practical applications for contents creation.