Whisper based Cross-Lingual Phoneme Recognition between Vietnamese and English
This addresses the challenge of accurate automatic speech recognition for mixed Vietnamese-English speech, which is incremental as it builds on existing pre-trained models.
The paper tackled cross-lingual phoneme recognition between Vietnamese and English by proposing a bilingual approach with a unified phoneme set and an end-to-end system using PhoWhisper, resulting in improved recognition accuracy for Vietnamese bilingual speech.
Cross-lingual phoneme recognition has emerged as a significant challenge for accurate automatic speech recognition (ASR) when mixing Vietnamese and English pronunciations. Unlike many languages, Vietnamese relies on tonal variations to distinguish word meanings, whereas English features stress patterns and non-standard pronunciations that hinder phoneme alignment between the two languages. To address this challenge, we propose a novel bilingual speech recognition approach with two primary contributions: (1) constructing a representative bilingual phoneme set that bridges the differences between Vietnamese and English phonetic systems; (2) designing an end-to-end system that leverages the PhoWhisper pre-trained encoder for deep high-level representations to improve phoneme recognition. Our extensive experiments demonstrate that the proposed approach not only improves recognition accuracy in bilingual speech recognition for Vietnamese but also provides a robust framework for addressing the complexities of tonal and stress-based phoneme recognition