Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems
This addresses incremental improvements for expanding multilingual ASR capabilities, which is relevant for speech recognition systems handling multiple languages.
The paper tackled the problem of sub-optimal continual learning for multilingual ASR by proposing four decoder optimizations, reducing the average word error rate for pretrained languages from 14.2% to 12.4% compared to Experience Replay while maintaining performance on new languages.
Continual Learning (CL) involves fine-tuning pre-trained models with new data while maintaining the performance on the pre-trained data. This is particularly relevant for expanding multilingual ASR (MASR) capabilities. However, existing CL methods, mainly designed for computer vision and reinforcement learning tasks, often yield sub-optimal results when directly applied to MASR. We hypothesise that this is because CL of the auto-regressive decoder in the MASR model is difficult. To verify this, we propose four optimizations on the decoder. They include decoder-layer gradient surgery, freezing unused token embeddings, suppressing output of newly added tokens, and learning rate re-scaling. Our experiments on adapting Whisper to 10 unseen languages from the Common Voice dataset demonstrate that these optimizations reduce the Average Word Error Rate (AWER) of pretrained languages from 14.2% to 12.4% compared with Experience Replay, without compromising the AWER of new languages.