V-Express: Conditional Dropout for Progressive Training of Portrait Video Generation
This addresses a challenge in portrait video generation for AI and multimedia applications, but it is incremental as it builds on existing generative models and adapters.
The paper tackles the problem of weak control signals being overshadowed by stronger ones in portrait video generation, particularly with audio, and proposes V-Express using progressive training and conditional dropout to balance these signals, achieving effective audio-controlled video generation as demonstrated in experiments.
In the field of portrait video generation, the use of single images to generate portrait videos has become increasingly prevalent. A common approach involves leveraging generative models to enhance adapters for controlled generation. However, control signals (e.g., text, audio, reference image, pose, depth map, etc.) can vary in strength. Among these, weaker conditions often struggle to be effective due to interference from stronger conditions, posing a challenge in balancing these conditions. In our work on portrait video generation, we identified audio signals as particularly weak, often overshadowed by stronger signals such as facial pose and reference image. However, direct training with weak signals often leads to difficulties in convergence. To address this, we propose V-Express, a simple method that balances different control signals through the progressive training and the conditional dropout operation. Our method gradually enables effective control by weak conditions, thereby achieving generation capabilities that simultaneously take into account the facial pose, reference image, and audio. The experimental results demonstrate that our method can effectively generate portrait videos controlled by audio. Furthermore, a potential solution is provided for the simultaneous and effective use of conditions of varying strengths.