SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation
This addresses the challenge of synthesizing realistic simultaneous human movements for applications in animation and robotics, representing an incremental advance in text-to-motion generation.
The paper tackles the problem of generating 3D human motions from text describing simultaneous actions, such as 'waving hand' while 'walking', by developing SINC, a method that uses GPT-3 to map actions to body parts and combines motions spatially, resulting in improved generation over baselines with synthetic data.
Our goal is to synthesize 3D human motions given textual inputs describing simultaneous actions, for example 'waving hand' while 'walking' at the same time. We refer to generating such simultaneous movements as performing 'spatial compositions'. In contrast to temporal compositions that seek to transition from one action to another, spatial compositing requires understanding which body parts are involved in which action, to be able to move them simultaneously. Motivated by the observation that the correspondence between actions and body parts is encoded in powerful language models, we extract this knowledge by prompting GPT-3 with text such as "what are the body parts involved in the action <action name>?", while also providing the parts list and few-shot examples. Given this action-part mapping, we combine body parts from two motions together and establish the first automated method to spatially compose two actions. However, training data with compositional actions is always limited by the combinatorics. Hence, we further create synthetic data with this approach, and use it to train a new state-of-the-art text-to-motion generation model, called SINC ("SImultaneous actioN Compositions for 3D human motions"). In our experiments, that training with such GPT-guided synthetic data improves spatial composition generation over baselines. Our code is publicly available at https://sinc.is.tue.mpg.de/.