FG-MDM: Towards Zero-Shot Human Motion Generation via ChatGPT-Refined Descriptions
This addresses a challenge in text-based motion generation for applications like animation or robotics, but it is incremental as it builds on existing diffusion models with refined descriptions.
The paper tackles the problem of generating human motions beyond the distribution of original datasets (zero-shot generation) by refining vague textual annotations into fine-grained descriptions using a large language model, and demonstrates superiority over previous methods in zero-shot settings.
Recently, significant progress has been made in text-based motion generation, enabling the generation of diverse and high-quality human motions that conform to textual descriptions. However, generating motions beyond the distribution of original datasets remains challenging, i.e., zero-shot generation. By adopting a divide-and-conquer strategy, we propose a new framework named Fine-Grained Human Motion Diffusion Model (FG-MDM) for zero-shot human motion generation. Specifically, we first parse previous vague textual annotations into fine-grained descriptions of different body parts by leveraging a large language model. We then use these fine-grained descriptions to guide a transformer-based diffusion model, which further adopts a design of part tokens. FG-MDM can generate human motions beyond the scope of original datasets owing to descriptions that are closer to motion essence. Our experimental results demonstrate the superiority of FG-MDM over previous methods in zero-shot settings. We will release our fine-grained textual annotations for HumanML3D and KIT.