FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles
This addresses the limitation of existing vision-driven TTS methods that rely on real-person faces, enabling broader applications with diverse characters and image styles, though it appears incremental.
The paper tackles the problem of generating speech from diverse portrait styles by introducing FaceSpeak, which extracts identity and emotional characteristics while filtering out irrelevant visual details, resulting in synthesized speech with satisfactory naturalness and quality.
Humans can perceive speakers' characteristics (e.g., identity, gender, personality and emotion) by their appearance, which are generally aligned to their voice style. Recently, vision-driven Text-to-speech (TTS) scholars grounded their investigations on real-person faces, thereby restricting effective speech synthesis from applying to vast potential usage scenarios with diverse characters and image styles. To solve this issue, we introduce a novel FaceSpeak approach. It extracts salient identity characteristics and emotional representations from a wide variety of image styles. Meanwhile, it mitigates the extraneous information (e.g., background, clothing, and hair color, etc.), resulting in synthesized speech closely aligned with a character's persona. Furthermore, to overcome the scarcity of multi-modal TTS data, we have devised an innovative dataset, namely Expressive Multi-Modal TTS, which is diligently curated and annotated to facilitate research in this domain. The experimental results demonstrate our proposed FaceSpeak can generate portrait-aligned voice with satisfactory naturalness and quality.