Advancing NAM-to-Speech Conversion with Novel Methods and the MultiNAM Dataset
This work addresses the challenge of generating intelligible speech from NAMs for applications like assistive technology, though it appears incremental by building on existing lip-to-speech and diffusion methods.
The paper tackles the problem of low intelligibility and poor generalization in Non-Audible Murmur (NAM)-to-speech conversion by proposing a diffusion-based method that incorporates lip modality and releases the MultiNAM dataset with 7.96 hours of paired data, achieving improved results as benchmarked on this dataset.
Current Non-Audible Murmur (NAM)-to-speech techniques rely on voice cloning to simulate ground-truth speech from paired whispers. However, the simulated speech often lacks intelligibility and fails to generalize well across different speakers. To address this issue, we focus on learning phoneme-level alignments from paired whispers and text and employ a Text-to-Speech (TTS) system to simulate the ground-truth. To reduce dependence on whispers, we learn phoneme alignments directly from NAMs, though the quality is constrained by the available training data. To further mitigate reliance on NAM/whisper data for ground-truth simulation, we propose incorporating the lip modality to infer speech and introduce a novel diffusion-based method that leverages recent advancements in lip-to-speech technology. Additionally, we release the MultiNAM dataset with over 7.96 hours of paired NAM, whisper, video, and text data from two speakers and benchmark all methods on this dataset. Speech samples and the dataset are available at https://diff-nam.github.io/DiffNAM/