Walk Before You Dance: High-fidelity and Editable Dance Synthesis via Generative Masked Motion Prior
This work addresses the problem of high-fidelity and editable dance synthesis for applications in animation, gaming, and virtual reality, representing a significant but incremental improvement over existing methods.
The paper tackles the challenge of generating realistic, synchronized, diverse, and physically plausible 3D dance motions by proposing a novel approach that uses a generative masked text-to-motion model as a prior to map guidance signals like music, genre, and pose into high-quality sequences, achieving state-of-the-art results in dance generation.
Recent advances in dance generation have enabled the automatic synthesis of 3D dance motions. However, existing methods still face significant challenges in simultaneously achieving high realism, precise dance-music synchronization, diverse motion expression, and physical plausibility. To address these limitations, we propose a novel approach that leverages a generative masked text-to-motion model as a distribution prior to learn a probabilistic mapping from diverse guidance signals, including music, genre, and pose, into high-quality dance motion sequences. Our framework also supports semantic motion editing, such as motion inpainting and body part modification. Specifically, we introduce a multi-tower masked motion model that integrates a text-conditioned masked motion backbone with two parallel, modality-specific branches: a music-guidance tower and a pose-guidance tower. The model is trained using synchronized and progressive masked training, which allows effective infusion of the pretrained text-to-motion prior into the dance synthesis process while enabling each guidance branch to optimize independently through its own loss function, mitigating gradient interference. During inference, we introduce classifier-free logits guidance and pose-guided token optimization to strengthen the influence of music, genre, and pose signals. Extensive experiments demonstrate that our method sets a new state of the art in dance generation, significantly advancing both the quality and editability over existing approaches. Project Page available at https://foram-s1.github.io/DanceMosaic/