Masked Acoustic Unit for Mispronunciation Detection and Correction
This addresses the problem of improving pronunciation training for language learners by offering a more efficient and effective feedback mechanism, though it appears incremental as it builds on existing CAPT approaches.
The paper tackles the limitations of conventional ASR-based Computer-Assisted Pronunciation Training (CAPT) methods, which require expensive annotation and provide only text-based feedback, by proposing a method using acoustic units (AUs) for both mispronunciation detection and correction, enabling speech-based self-imitating feedback.
Computer-Assisted Pronunciation Training (CAPT) plays an important role in language learning. Conventional ASR-based CAPT methods require expensive annotation of the ground truth pronunciation for the supervised training. Meanwhile, certain undefined non-native phonemes cannot be correctly classified into standard phonemes, making the annotation process challenging and subjective. On the other hand, ASR-based CAPT methods only give the learner text-based feedback about the mispronunciation, but cannot teach the learner how to pronounce the sentence correctly. To solve these limitations, we propose to use the acoustic unit (AU) as the intermediary feature for both mispronunciation detection and correction. The proposed method uses the masked AU sequence and the target phonemes to detect the error AU and then corrects it. This method can give the learner speech-based self-imitating feedback, making our CAPT powerful for education.