MRI2Speech: Speech Synthesis from Articulatory Movements Recorded by Real-time MRI
This work addresses speech synthesis for individuals with speech impairments by enabling voice generation from MRI data, representing a strong specific gain in this domain.
The paper tackles the problem of synthesizing speech from articulatory movements recorded by real-time MRI, which previously suffered from poor intelligibility due to noise entanglement, and achieves a 15.18% Word Error Rate on the USC-TIMIT MRI corpus, a significant improvement over the state-of-the-art.
Previous real-time MRI (rtMRI)-based speech synthesis models depend heavily on noisy ground-truth speech. Applying loss directly over ground truth mel-spectrograms entangles speech content with MRI noise, resulting in poor intelligibility. We introduce a novel approach that adapts the multi-modal self-supervised AV-HuBERT model for text prediction from rtMRI and incorporates a new flow-based duration predictor for speaker-specific alignment. The predicted text and durations are then used by a speech decoder to synthesize aligned speech in any novel voice. We conduct thorough experiments on two datasets and demonstrate our method's generalization ability to unseen speakers. We assess our framework's performance by masking parts of the rtMRI video to evaluate the impact of different articulators on text prediction. Our method achieves a $15.18\%$ Word Error Rate (WER) on the USC-TIMIT MRI corpus, marking a huge improvement over the current state-of-the-art. Speech samples are available at https://mri2speech.github.io/MRI2Speech/