Language-Aware Prompt Tuning for Parameter-Efficient Seamless Language Expansion in Multilingual ASR
It addresses efficient language expansion for multilingual ASR systems, offering incremental improvements with minimal computational overhead.
The paper tackled language interference and expansion in multilingual ASR by proposing Entire SPT and LAPT methods, which improved performance by 5.0% and 16.0% over baseline in language expansion tasks on FLEURS data.
Recent advancements in multilingual automatic speech recognition (ASR) have been driven by large-scale end-to-end models like Whisper. However, challenges such as language interference and expanding to unseen languages (language expansion) without degrading performance persist. This paper addresses these with three contributions: 1) Entire Soft Prompt Tuning (Entire SPT), which applies soft prompts to both the encoder and decoder, enhancing feature extraction and decoding; 2) Language-Aware Prompt Tuning (LAPT), which leverages cross-lingual similarities to encode shared and language-specific features using lightweight prompt matrices; 3) SPT-Whisper, a toolkit that integrates SPT into Whisper and enables efficient continual learning. Experiments across three languages from FLEURS demonstrate that Entire SPT and LAPT outperform Decoder SPT by 5.0% and 16.0% in language expansion tasks, respectively, providing an efficient solution for dynamic, multilingual ASR models with minimal computational overhead.