Freeform Body Motion Generation from Speech
This work improves co-speech motion generation for applications like virtual avatars and human-computer interaction, though it is incremental by building on existing methods with a novel decomposition approach.
The paper tackles the problem of generating diverse and natural body motions from speech by addressing the non-deterministic mapping, introducing a two-stream model that decomposes motion into pose modes and rhythmic dynamics. It demonstrates superior performance in motion diversity, quality, and speech syncing compared to baselines.
People naturally conduct spontaneous body motions to enhance their speeches while giving talks. Body motion generation from speech is inherently difficult due to the non-deterministic mapping from speech to body motions. Most existing works map speech to motion in a deterministic way by conditioning on certain styles, leading to sub-optimal results. Motivated by studies in linguistics, we decompose the co-speech motion into two complementary parts: pose modes and rhythmic dynamics. Accordingly, we introduce a novel freeform motion generation model (FreeMo) by equipping a two-stream architecture, i.e., a pose mode branch for primary posture generation, and a rhythmic motion branch for rhythmic dynamics synthesis. On one hand, diverse pose modes are generated by conditional sampling in a latent space, guided by speech semantics. On the other hand, rhythmic dynamics are synced with the speech prosody. Extensive experiments demonstrate the superior performance against several baselines, in terms of motion diversity, quality and syncing with speech. Code and pre-trained models will be publicly available through https://github.com/TheTempAccount/Co-Speech-Motion-Generation.