PiDA: Phonetically-Informed Data Augmentation for Robust Vietnamese Speech Translation
For developers of Vietnamese speech translation systems, this work provides a method to mitigate ASR error propagation, though the approach is incremental.
The paper systematically categorizes ASR errors in Vietnamese speech translation, finding that most substitution errors are due to phonetic confusions. They propose PiDA, a data augmentation method that generates phonetic corruptions, improving translation of erroneous ASR outputs by up to +2.04 BLEU while also slightly improving clean-text performance.
Cascaded speech translation (ST) systems suffer from error propagation when Automatic Speech Recognition (ASR) outputs incorrect transcripts. We present the first systematic categorization of ASR errors for Vietnamese ST, classifying substitution errors by phonetic cause and quantifying their impact on downstream Neural Machine Translation (NMT) performance using Linear Mixed-Effects Modelling. We confirm that most ASR substitution errors arise from phonetic confusions rather than random noise, and that these phonetic errors significantly degrade ST quality. Motivated by this finding, we propose Phonetically-Informed Data Augmentation (PiDA), which generates ASR-like corruptions by substituting words with phonetically similar alternatives using phonetic word embeddings. Fine-tuning on a PiDA-augmented version of FLEURS Vietnamese-English improves translation of erroneous ASR outputs (up to +2.04 BLEU over standard fine-tuning) while also slightly improving clean-text performance.