Building High-accuracy Multilingual ASR with Gated Language Experts and Curriculum Training
This addresses the challenge of accurate multilingual ASR for users in diverse language settings, representing a strong incremental improvement over existing methods.
The paper tackled the problem of building multilingual automatic speech recognition (ASR) without requiring language identification input during inference, achieving average relative word error reductions of 12.5% and 7.3% compared to baseline bilingual and monolingual models, respectively, with performance comparable to an oracle model.
We propose gated language experts and curriculum training to enhance multilingual transformer transducer models without requiring language identification (LID) input from users during inference. Our method incorporates a gating mechanism and LID loss, enabling transformer experts to learn language-specific information. By combining gated transformer experts with shared transformer layers, we construct multilingual transformer blocks and utilize linear experts to effectively regularize the joint network. The curriculum training scheme leverages LID to guide the gated experts in improving their respective language performance. Experimental results on a bilingual task involving English and Spanish demonstrate significant improvements, with average relative word error reductions of 12.5% and 7.3% compared to the baseline bilingual and monolingual models, respectively. Notably, our method achieves performance comparable to the upper-bound model trained and inferred with oracle LID. Extending our approach to trilingual, quadrilingual, and pentalingual models reveals similar advantages to those observed in the bilingual models, highlighting its ease of extension to multiple languages.