Every Image Listens, Every Image Dances: Music-Driven Image Animation
This addresses the problem of generating personalized, synchronized dance videos for users of all expertise levels, representing a novel but domain-specific advancement.
The paper tackles music-driven dance video generation by introducing MuseDance, an end-to-end model that animates images using music and text inputs, eliminating the need for complex motion guidance and achieving robust generalization and temporal consistency.
Image animation has become a promising area in multimodal research, with a focus on generating videos from reference images. While prior work has largely emphasized generic video generation guided by text, music-driven dance video generation remains underexplored. In this paper, we introduce MuseDance, an innovative end-to-end model that animates reference images using both music and text inputs. This dual input enables MuseDance to generate personalized videos that follow text descriptions and synchronize character movements with the music. Unlike existing approaches, MuseDance eliminates the need for complex motion guidance inputs, such as pose or depth sequences, making flexible and creative video generation accessible to users of all expertise levels. To advance research in this field, we present a new multimodal dataset comprising 2,904 dance videos with corresponding background music and text descriptions. Our approach leverages diffusion-based methods to achieve robust generalization, precise control, and temporal consistency, setting a new baseline for the music-driven image animation task.