MAGMA: Music Aligned Generative Motion Autodecoder
This addresses the challenge of creating coherent and synchronized dance animations from music, which is useful for entertainment and animation industries, though it appears incremental by building on existing methods like VQ-VAEs and Transformers.
The paper tackles the problem of generating dance motions synchronized with music by introducing a two-step approach using a VQ-VAE and Transformer decoder, achieving state-of-the-art results in benchmarks and enabling real-time generation of longer, customizable sequences.
Mapping music to dance is a challenging problem that requires spatial and temporal coherence along with a continual synchronization with the music's progression. Taking inspiration from large language models, we introduce a 2-step approach for generating dance using a Vector Quantized-Variational Autoencoder (VQ-VAE) to distill motion into primitives and train a Transformer decoder to learn the correct sequencing of these primitives. We also evaluate the importance of music representations by comparing naive music feature extraction using Librosa to deep audio representations generated by state-of-the-art audio compression algorithms. Additionally, we train variations of the motion generator using relative and absolute positional encodings to determine the effect on generated motion quality when generating arbitrarily long sequence lengths. Our proposed approach achieve state-of-the-art results in music-to-motion generation benchmarks and enables the real-time generation of considerably longer motion sequences, the ability to chain multiple motion sequences seamlessly, and easy customization of motion sequences to meet style requirements.