Improving generalization of vocal tract feature reconstruction: from augmented acoustic inversion to articulatory feature reconstruction without articulatory data
This work addresses the challenge of articulatory feature reconstruction for speech processing, particularly in scenarios with limited data, offering a method that generalizes across speakers and datasets.
The paper tackles the problem of reconstructing articulatory movements from audio or phonetic labels, addressing generalization issues due to scarce multi-speaker data. It shows that phonetic labels outperform acoustic features in reconstruction tasks and introduces a novel method that achieves up to 0.59 Pearson correlation with measured articulatory features without using articulatory data.
We address the problem of reconstructing articulatory movements, given audio and/or phonetic labels. The scarce availability of multi-speaker articulatory data makes it difficult to learn a reconstruction that generalizes to new speakers and across datasets. We first consider the XRMB dataset where audio, articulatory measurements and phonetic transcriptions are available. We show that phonetic labels, used as input to deep recurrent neural networks that reconstruct articulatory features, are in general more helpful than acoustic features in both matched and mismatched training-testing conditions. In a second experiment, we test a novel approach that attempts to build articulatory features from prior articulatory information extracted from phonetic labels. Such approach recovers vocal tract movements directly from an acoustic-only dataset without using any articulatory measurement. Results show that articulatory features generated by this approach can correlate up to 0.59 Pearson product-moment correlation with measured articulatory features.