JoyStreamer: Unlocking Highly Expressive Avatars via Harmonized Text-Audio Conditioning
This work addresses the challenge of generating highly expressive avatars for applications such as multi-person dialogues and role-playing, representing a significant advancement over prior methods.
The paper tackles the problem of limited text instruction alignment in existing video avatar models for complex elements like full-body movement and dynamic camera trajectories, resulting in JoyAvatar, which outperforms state-of-the-art models like Omnihuman-1.5 and KlingAvatar 2.0 in generating long-duration avatar videos with natural motions and preserved capabilities.
Existing video avatar models have demonstrated impressive capabilities in scenarios such as talking, public speaking, and singing. However, the majority of these methods exhibit limited alignment with respect to text instructions, particularly when the prompts involve complex elements including large full-body movement, dynamic camera trajectory, background transitions, or human-object interactions. To break out this limitation, we present JoyAvatar, a framework capable of generating long duration avatar videos, featuring two key technical innovations. Firstly, we introduce a twin-teacher enhanced training algorithm that enables the model to transfer inherent text-controllability from the foundation model while simultaneously learning audio-visual synchronization. Secondly, during training, we dynamically modulate the strength of multi-modal conditions (e.g., audio and text) based on the distinct denoising timestep, aiming to mitigate conflicts between the heterogeneous conditioning signals. These two key designs serve to substantially expand the avatar model's capacity to generate natural, temporally coherent full-body motions and dynamic camera movements as well as preserve the basic avatar capabilities, such as accurate lip-sync and identity consistency. GSB evaluation results demonstrate that our JoyStreamer model outperforms the state-of-the-art models such as Omnihuman-1.5 and KlingAvatar 2.0. Moreover, our approach enables complex applications including multi-person dialogues and non-human subjects role-playing. Some video samples are provided on https://joystreamer.github.io/.