TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation
This work addresses the need for human-like conversational systems by integrating previously isolated audio and visual generation tasks, though it is incremental in combining existing methods.
The paper tackled the problem of jointly synthesizing interactive videos and conversational speech from text and reference images, resulting in a unified framework that effectively generates synchronized audio-visual interactions across multiple evaluation dimensions.
The objective of this paper is to jointly synthesize interactive videos and conversational speech from text and reference images. With the ultimate goal of building human-like conversational systems, recent studies have explored talking or listening head generation as well as conversational speech generation. However, these works are typically studied in isolation, overlooking the multimodal nature of human conversation, which involves tightly coupled audio-visual interactions. In this paper, we introduce TAVID, a unified framework that generates both interactive faces and conversational speech in a synchronized manner. TAVID integrates face and speech generation pipelines through two cross-modal mappers (i.e., a motion mapper and a speaker mapper), which enable bidirectional exchange of complementary information between the audio and visual modalities. We evaluate our system across four dimensions: talking face realism, listening head responsiveness, dyadic interaction fluency, and speech quality. Extensive experiments demonstrate the effectiveness of our approach across all these aspects.