English Pronunciation Evaluation without Complex Joint Training: LoRA Fine-tuned Speech Multimodal LLM
This provides a simpler, more accessible method for Computer-Assisted Pronunciation Training (CAPT) technologies, though it is incremental as it adapts existing models rather than introducing a fundamentally new approach.
This study tackled the problem of simultaneously performing Automatic Pronunciation Assessment (APA) and Mispronunciation Detection and Diagnosis (MDD) for English L2 learners by fine-tuning a Multimodal Large Language Model (MLLM) using Low-Rank Adaptation (LoRA), achieving a Pearson Correlation Coefficient > 0.7 with human scores and Word/Phoneme Error Rates < 0.15.
This study demonstrates that a Multimodal Large Language Model (MLLM) adapted via Low-Rank Adaptation (LoRA) can perform both Automatic Pronunciation Assessment (APA) and Mispronunciation Detection and Diagnosis (MDD) simultaneously. Leveraging Microsoft's Phi-4-multimodal-instruct, our fine-tuning method eliminates the need for complex architectural changes or separate training procedures conventionally required for these distinct tasks. Fine-tuned on the Speechocean762 dataset, the pronunciation evaluation scores predicted by the model exhibited a strong Pearson Correlation Coefficient (PCC > 0.7) with human-assigned scores, while achieving low Word Error Rate (WER) and Phoneme Error Rate (PER) (both < 0.15). Notably, fine-tuning only the LoRA layers was sufficient to achieve performance levels comparable to those achieved by fine-tuning all audio layers. This research highlights that an integrated pronunciation assessment system can be established by adapting large multimodal models without full fine-tuning, utilizing a significantly simpler training methodology compared to previous joint models designed for simultaneous APA and MDD. This efficient LoRA-based approach paves the way for more accessible, integrated, and effective Computer-Assisted Pronunciation Training (CAPT) technologies for English L2 learners.