Extending Text-to-Speech Synthesis with Articulatory Movement Prediction using Ultrasound Tongue Imaging
This work addresses audiovisual speech synthesis and pronunciation training, but it is incremental as it builds on existing TTS methods with a new articulatory component.
The paper tackled the problem of predicting articulatory tongue movements from text for speech synthesis by extending a DNN-TTS framework to predict ultrasound tongue images, showing that fully connected DNNs outperform LSTMs with limited data and generating videos close to natural movement.
In this paper, we present our first experiments in text-to-articulation prediction, using ultrasound tongue image targets. We extend a traditional (vocoder-based) DNN-TTS framework with predicting PCA-compressed ultrasound images, of which the continuous tongue motion can be reconstructed in synchrony with synthesized speech. We use the data of eight speakers, train fully connected and recurrent neural networks, and show that FC-DNNs are more suitable for the prediction of sequential data than LSTMs, in case of limited training data. Objective experiments and visualized predictions show that the proposed solution is feasible and the generated ultrasound videos are close to natural tongue movement. Articulatory movement prediction from text input can be useful for audiovisual speech synthesis or computer-assisted pronunciation training.