Speech2Video: Cross-Modal Distillation for Speech to Video Generation
This work addresses a novel task in cross-modal generation for applications in entertainment and human-computer interaction, representing an incremental advancement over prior methods.
The paper tackles the problem of generating talking face videos from speech alone by proposing a cross-modal distillation method to disentangle emotional and identity information from audio, achieving results that are almost indistinguishable from baseline methods and outperforming existing algorithms in emotion expression.
This paper investigates a novel task of talking face video generation solely from speeches. The speech-to-video generation technique can spark interesting applications in entertainment, customer service, and human-computer-interaction industries. Indeed, the timbre, accent and speed in speeches could contain rich information relevant to speakers' appearance. The challenge mainly lies in disentangling the distinct visual attributes from audio signals. In this article, we propose a light-weight, cross-modal distillation method to extract disentangled emotional and identity information from unlabelled video inputs. The extracted features are then integrated by a generative adversarial network into talking face video clips. With carefully crafted discriminators, the proposed framework achieves realistic generation results. Experiments with observed individuals demonstrated that the proposed framework captures the emotional expressions solely from speeches, and produces spontaneous facial motion in the video output. Compared to the baseline method where speeches are combined with a static image of the speaker, the results of the proposed framework is almost indistinguishable. User studies also show that the proposed method outperforms the existing algorithms in terms of emotion expression in the generated videos.