Generating Human Motion Videos using a Cascaded Text-to-Video Framework
This work addresses the underexplored task of general-purpose human video generation for applications in graphics, entertainment, and embodied AI, representing an incremental advancement by integrating existing components with novel conditioning modules.
The paper tackled the problem of generating human motion videos from text by proposing CAMEO, a cascaded framework that bridges Text-to-Motion models and conditional video diffusion models, achieving robust alignment and enhanced coherence across diverse benchmarks.
Human video generation is becoming an increasingly important task with broad applications in graphics, entertainment, and embodied AI. Despite the rapid progress of video diffusion models (VDMs), their use for general-purpose human video generation remains underexplored, with most works constrained to image-to-video setups or narrow domains like dance videos. In this work, we propose CAMEO, a cascaded framework for general human motion video generation. It seamlessly bridges Text-to-Motion (T2M) models and conditional VDMs, mitigating suboptimal factors that may arise in this process across both training and inference through carefully designed components. Specifically, we analyze and prepare both textual prompts and visual conditions to effectively train the VDM, ensuring robust alignment between motion descriptions, conditioning signals, and the generated videos. Furthermore, we introduce a camera-aware conditioning module that connects the two stages, automatically selecting viewpoints aligned with the input text to enhance coherence and reduce manual intervention. We demonstrate the effectiveness of our approach on both the MovieGen benchmark and a newly introduced benchmark tailored to the T2M-VDM combination, while highlighting its versatility across diverse use cases.