KeyVID: Keyframe-Aware Video Diffusion for Audio-Synchronized Visual Animation
This work solves the problem of inefficient and low-quality audio-to-visual animation for applications like video generation, though it is incremental as it builds on existing diffusion models.
The paper tackles the problem of generating video from audio by addressing inefficiencies in uniformly sampled frames for dramatic motions, proposing KeyVID to localize and generate keyframes from audio, which improves audio-video synchronization and quality, with significant gains demonstrated in experiments.
Generating video from various conditions, such as text, image, and audio, enables both spatial and temporal control, leading to high-quality generation results. Videos with dramatic motions often require a higher frame rate to ensure smooth motion. Currently, most audio-to-visual animation models use uniformly sampled frames from video clips. However, these uniformly sampled frames fail to capture significant key moments in dramatic motions at low frame rates and require significantly more memory when increasing the number of frames directly. In this paper, we propose KeyVID, a keyframe-aware audio-to-visual animation framework that significantly improves the generation quality for key moments in audio signals while maintaining computation efficiency. Given an image and an audio input, we first localize keyframe time steps from the audio. Then, we use a keyframe generator to generate the corresponding visual keyframes. Finally, we generate all intermediate frames using the motion interpolator. Through extensive experiments, we demonstrate that KeyVID significantly improves audio-video synchronization and video quality across multiple datasets, particularly for highly dynamic motions. The code is released in https://github.com/XingruiWang/KeyVID.