Residual-guided Personalized Speech Synthesis based on Face Image
This addresses the problem of data scarcity for personalized speech synthesis, offering a more efficient approach for applications like virtual assistants or accessibility tools, though it is incremental as it builds on existing neural vocoder and face-speech linkage concepts.
The paper tackles personalized speech synthesis by extracting speech features from face images instead of requiring large audio datasets, achieving comparable quality to previous audio-based methods.
Previous works derive personalized speech features by training the model on a large dataset composed of his/her audio sounds. It was reported that face information has a strong link with the speech sound. Thus in this work, we innovatively extract personalized speech features from human faces to synthesize personalized speech using neural vocoder. A Face-based Residual Personalized Speech Synthesis Model (FR-PSS) containing a speech encoder, a speech synthesizer and a face encoder is designed for PSS. In this model, by designing two speech priors, a residual-guided strategy is introduced to guide the face feature to approach the true speech feature in the training. Moreover, considering the error of feature's absolute values and their directional bias, we formulate a novel tri-item loss function for face encoder. Experimental results show that the speech synthesized by our model is comparable to the personalized speech synthesized by training a large amount of audio data in previous works.