Enhancing Motion Variation in Text-to-Motion Models via Pose and Video Conditioned Editing
This addresses the problem of motion diversity for users of text-to-motion models, offering an incremental improvement by enhancing existing models with additional modalities.
The paper tackles the limited motion variation in text-to-motion models due to data scarcity by proposing a method that uses short video clips or images as conditions to edit basic motions, enabling generation of unseen motions like kicking a football with the instep, with a user study showing realism comparable to common motions.
Text-to-motion models that generate sequences of human poses from textual descriptions are garnering significant attention. However, due to data scarcity, the range of motions these models can produce is still limited. For instance, current text-to-motion models cannot generate a motion of kicking a football with the instep of the foot, since the training data only includes martial arts kicks. We propose a novel method that uses short video clips or images as conditions to modify existing basic motions. In this approach, the model's understanding of a kick serves as the prior, while the video or image of a football kick acts as the posterior, enabling the generation of the desired motion. By incorporating these additional modalities as conditions, our method can create motions not present in the training set, overcoming the limitations of text-motion datasets. A user study with 26 participants demonstrated that our approach produces unseen motions with realism comparable to commonly represented motions in text-motion datasets (e.g., HumanML3D), such as walking, running, squatting, and kicking.