Speaker dependent acoustic-to-articulatory inversion using real-time MRI of the vocal tract
This work addresses speech processing tasks like synthesis and recognition by providing detailed articulatory data, but it is incremental as it applies existing neural network methods to a new imaging modality.
The study tackled speaker-dependent acoustic-to-articulatory inversion by estimating midsagittal real-time MRI images of the vocal tract from acoustic features, achieving an average CW-SSIM of 0.94 for generated images similar to original recordings.
Acoustic-to-articulatory inversion (AAI) methods estimate articulatory movements from the acoustic speech signal, which can be useful in several tasks such as speech recognition, synthesis, talking heads and language tutoring. Most earlier inversion studies are based on point-tracking articulatory techniques (e.g. EMA or XRMB). The advantage of rtMRI is that it provides dynamic information about the full midsagittal plane of the upper airway, with a high 'relative' spatial resolution. In this work, we estimated midsagittal rtMRI images of the vocal tract for speaker dependent AAI, using MGC-LSP spectral features as input. We applied FC-DNNs, CNNs and recurrent neural networks, and have shown that LSTMs are the most suitable for this task. As objective evaluation we measured normalized MSE, Structural Similarity Index (SSIM) and its complex wavelet version (CW-SSIM). The results indicate that the combination of FC-DNNs and LSTMs can achieve smooth generated MR images of the vocal tract, which are similar to the original MRI recordings (average CW-SSIM: 0.94).