Action-GPT: Leveraging Large-scale Language Models for Improved and Generalized Action Generation
This work addresses the challenge of generating realistic motions from text for applications in animation and robotics, but it is incremental as it builds on existing text-to-motion models by enhancing input descriptions.
The authors tackled the problem of text-based action generation by using Large Language Models (LLMs) to create richer descriptions from minimal action phrases, resulting in improved alignment between text and motion spaces and better synthesized motions with zero-shot capabilities.
We introduce Action-GPT, a plug-and-play framework for incorporating Large Language Models (LLMs) into text-based action generation models. Action phrases in current motion capture datasets contain minimal and to-the-point information. By carefully crafting prompts for LLMs, we generate richer and fine-grained descriptions of the action. We show that utilizing these detailed descriptions instead of the original action phrases leads to better alignment of text and motion spaces. We introduce a generic approach compatible with stochastic (e.g. VAE-based) and deterministic (e.g. MotionCLIP) text-to-motion models. In addition, the approach enables multiple text descriptions to be utilized. Our experiments show (i) noticeable qualitative and quantitative improvement in the quality of synthesized motions, (ii) benefits of utilizing multiple LLM-generated descriptions, (iii) suitability of the prompt function, and (iv) zero-shot generation capabilities of the proposed approach. Project page: https://actiongpt.github.io