MIDGET: Music Conditioned 3D Dance Generation
This addresses the challenge of creating realistic, music-synchronized dance animations for applications in entertainment and virtual reality, representing an incremental improvement over existing methods.
The paper tackles the problem of generating 3D dance motions that align with music rhythm by proposing MIDGET, a model based on VQ-VAE and GPT, which achieves state-of-the-art performance on motion quality and music alignment on the AIST++ dataset.
In this paper, we introduce a MusIc conditioned 3D Dance GEneraTion model, named MIDGET based on Dance motion Vector Quantised Variational AutoEncoder (VQ-VAE) model and Motion Generative Pre-Training (GPT) model to generate vibrant and highquality dances that match the music rhythm. To tackle challenges in the field, we introduce three new components: 1) a pre-trained memory codebook based on the Motion VQ-VAE model to store different human pose codes, 2) employing Motion GPT model to generate pose codes with music and motion Encoders, 3) a simple framework for music feature extraction. We compare with existing state-of-the-art models and perform ablation experiments on AIST++, the largest publicly available music-dance dataset. Experiments demonstrate that our proposed framework achieves state-of-the-art performance on motion quality and its alignment with the music.