OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation
This work addresses the need for efficient and controllable audio-driven full-body animation for applications like podcasts and singing, representing an incremental improvement over prior methods.
The paper tackles the problem of generating full-body avatar videos from audio, addressing limitations in existing methods that focus on facial movements and lack precise control, by introducing OmniAvatar, which achieves improved lip-sync accuracy and natural movements, surpassing existing models in facial and semi-body video generation.
Significant progress has been made in audio-driven human animation, while most existing methods focus mainly on facial movements, limiting their ability to create full-body animations with natural synchronization and fluidity. They also struggle with precise prompt control for fine-grained generation. To tackle these challenges, we introduce OmniAvatar, an innovative audio-driven full-body video generation model that enhances human animation with improved lip-sync accuracy and natural movements. OmniAvatar introduces a pixel-wise multi-hierarchical audio embedding strategy to better capture audio features in the latent space, enhancing lip-syncing across diverse scenes. To preserve the capability for prompt-driven control of foundation models while effectively incorporating audio features, we employ a LoRA-based training approach. Extensive experiments show that OmniAvatar surpasses existing models in both facial and semi-body video generation, offering precise text-based control for creating videos in various domains, such as podcasts, human interactions, dynamic scenes, and singing. Our project page is https://omni-avatar.github.io/.