DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation
This work addresses the challenge of automatic dance generation for applications in entertainment and animation, but it is incremental as it builds upon existing diffusion and contrastive learning techniques.
The paper tackles the problem of generating realistic dance sequences from music by addressing compounding errors and long-term structure issues in autoregressive methods, resulting in a cascaded diffusion model that produces high-resolution, long-form dances comparable to state-of-the-art methods on the AIST++ dataset.
When hearing music, it is natural for people to dance to its rhythm. Automatic dance generation, however, is a challenging task due to the physical constraints of human motion and rhythmic alignment with target music. Conventional autoregressive methods introduce compounding errors during sampling and struggle to capture the long-term structure of dance sequences. To address these limitations, we present a novel cascaded motion diffusion model, DiffDance, designed for high-resolution, long-form dance generation. This model comprises a music-to-dance diffusion model and a sequence super-resolution diffusion model. To bridge the gap between music and motion for conditional generation, DiffDance employs a pretrained audio representation learning model to extract music embeddings and further align its embedding space to motion via contrastive loss. During training our cascaded diffusion model, we also incorporate multiple geometric losses to constrain the model outputs to be physically plausible and add a dynamic loss weight that adaptively changes over diffusion timesteps to facilitate sample diversity. Through comprehensive experiments performed on the benchmark dataset AIST++, we demonstrate that DiffDance is capable of generating realistic dance sequences that align effectively with the input music. These results are comparable to those achieved by state-of-the-art autoregressive methods.