EnergyMoGen: Compositional Human Motion Generation with Energy-Based Diffusion Model in Latent Space
This work addresses the problem of generating complex, multi-concept human motions for applications in animation and robotics, representing an incremental improvement over existing latent diffusion methods.
The paper tackles the challenge of composing multiple semantic concepts into coherent human motion sequences using latent diffusion models, proposing EnergyMoGen with Energy-Based Models and Synergistic Energy Fusion to outperform state-of-the-art models in tasks like text-to-motion and compositional motion generation.
Diffusion models, particularly latent diffusion models, have demonstrated remarkable success in text-driven human motion generation. However, it remains challenging for latent diffusion models to effectively compose multiple semantic concepts into a single, coherent motion sequence. To address this issue, we propose EnergyMoGen, which includes two spectrums of Energy-Based Models: (1) We interpret the diffusion model as a latent-aware energy-based model that generates motions by composing a set of diffusion models in latent space; (2) We introduce a semantic-aware energy model based on cross-attention, which enables semantic composition and adaptive gradient descent for text embeddings. To overcome the challenges of semantic inconsistency and motion distortion across these two spectrums, we introduce Synergistic Energy Fusion. This design allows the motion latent diffusion model to synthesize high-quality, complex motions by combining multiple energy terms corresponding to textual descriptions. Experiments show that our approach outperforms existing state-of-the-art models on various motion generation tasks, including text-to-motion generation, compositional motion generation, and multi-concept motion generation. Additionally, we demonstrate that our method can be used to extend motion datasets and improve the text-to-motion task.