OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Models
This work addresses the scalability and flexibility limitations in end-to-end human animation for applications like audio-driven talking human generation, offering a more versatile and realistic solution.
The paper tackles the problem of scaling up one-stage conditioned human animation models, which struggle to match large general video generation models, by proposing OmniHuman, a Diffusion Transformer-based framework that mixes motion-related conditions during training, achieving highly realistic human video generation with support for various portrait contents, talking and singing, human-object interactions, and multiple driving modalities.
End-to-end human animation, such as audio-driven talking human generation, has undergone notable advancements in the recent few years. However, existing methods still struggle to scale up as large general video generation models, limiting their potential in real applications. In this paper, we propose OmniHuman, a Diffusion Transformer-based framework that scales up data by mixing motion-related conditions into the training phase. To this end, we introduce two training principles for these mixed conditions, along with the corresponding model architecture and inference strategy. These designs enable OmniHuman to fully leverage data-driven motion generation, ultimately achieving highly realistic human video generation. More importantly, OmniHuman supports various portrait contents (face close-up, portrait, half-body, full-body), supports both talking and singing, handles human-object interactions and challenging body poses, and accommodates different image styles. Compared to existing end-to-end audio-driven methods, OmniHuman not only produces more realistic videos, but also offers greater flexibility in inputs. It also supports multiple driving modalities (audio-driven, video-driven and combined driving signals). Video samples are provided on the ttfamily project page (https://omnihuman-lab.github.io)