An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data
This work addresses acoustic-to-articulatory inversion for speech analysis and clinical applications, representing an incremental improvement by combining audio and text inputs to enhance UTI data generation.
The paper tackled the problem of generating high-quality ultrasound tongue imaging (UTI) data from speech signals by proposing an audio-textual diffusion model that encodes individual acoustic characteristics and ASR transcriptions, resulting in clear tongue contours crucial for linguistic analysis and clinical assessment.
Acoustic-to-articulatory inversion (AAI) is to convert audio into articulator movements, such as ultrasound tongue imaging (UTI) data. An issue of existing AAI methods is only using the personalized acoustic information to derive the general patterns of tongue motions, and thus the quality of generated UTI data is limited. To address this issue, this paper proposes an audio-textual diffusion model for the UTI data generation task. In this model, the inherent acoustic characteristics of individuals related to the tongue motion details are encoded by using wav2vec 2.0, while the ASR transcriptions related to the universality of tongue motions are encoded by using BERT. UTI data are then generated by using a diffusion module. Experimental results showed that the proposed diffusion model could generate high-quality UTI data with clear tongue contour that is crucial for the linguistic analysis and clinical assessment. The project can be found on the website\footnote{https://yangyudong2020.github.io/wav2uti/