LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR
This addresses the problem of efficient and scalable multilingual ASR for users needing to handle multiple languages without performance loss, representing an incremental improvement.
The paper tackles language interference and performance degradation when adding new languages in multilingual automatic speech recognition by proposing LoRA-Whisper, which incorporates LoRA into Whisper, resulting in relative gains of 18.5% for multilingual ASR and 23.0% for language expansion over the baseline.
Recent years have witnessed significant progress in multilingual automatic speech recognition (ASR), driven by the emergence of end-to-end (E2E) models and the scaling of multilingual datasets. Despite that, two main challenges persist in multilingual ASR: language interference and the incorporation of new languages without degrading the performance of the existing ones. This paper proposes LoRA-Whisper, which incorporates LoRA matrix into Whisper for multilingual ASR, effectively mitigating language interference. Furthermore, by leveraging LoRA and the similarities between languages, we can achieve better performance on new languages while upholding consistent performance on original ones. Experiments on a real-world task across eight languages demonstrate that our proposed LoRA-Whisper yields a relative gain of 18.5% and 23.0% over the baseline system for multilingual ASR and language expansion respectively.