Unsupervised Rhythm and Voice Conversion to Improve ASR on Dysarthric Speech
This addresses ASR performance for dysarthric speakers, but it is incremental as it extends an existing framework.
The paper tackled the problem of ASR struggling with dysarthric speech by exploring dysarthric-to-healthy speech conversion, resulting in significant word error rate reductions for LF-MMI models, especially in severe cases, while fine-tuning Whisper had minimal effect.
Automatic speech recognition (ASR) systems struggle with dysarthric speech due to high inter-speaker variability and slow speaking rates. To address this, we explore dysarthric-to-healthy speech conversion for improved ASR performance. Our approach extends the Rhythm and Voice (RnV) conversion framework by introducing a syllable-based rhythm modeling method suited for dysarthric speech. We assess its impact on ASR by training LF-MMI models and fine-tuning Whisper on converted speech. Experiments on the Torgo corpus reveal that LF-MMI achieves significant word error rate reductions, especially for more severe cases of dysarthria, while fine-tuning Whisper on converted data has minimal effect on its performance. These results highlight the potential of unsupervised rhythm and voice conversion for dysarthric ASR. Code available at: https://github.com/idiap/RnV