Lamer-SSL: Layer-aware Mixture of LoRA Experts for Continual Multilingual Expansion of Self-supervised Models without Forgetting
This addresses the challenge of continual multilingual expansion for self-supervised speech models, offering a parameter-efficient solution to mitigate forgetting, though it is incremental as it builds on existing methods like LoRA and replay strategies.
The paper tackled the problem of self-supervised speech models struggling to generalize to new languages and forgetting prior knowledge during continual training, proposing Lamer-SSL, which achieved effective multilingual expansion with only 2.14% trainable parameters while maintaining strong performance on previously learned languages.
Despite their impressive performance, self-supervised speech models often struggle to generalize to new languages and tend to forget previously acquired knowledge during continual training. To address this, we propose Lamer-SSL, a parameter-efficient framework that integrates a Layer-Aware MixturE of LoRA Experts (Lamer) module with a replay strategy. The Lamer module enables flexible balancing between shared and language-specific representations, while layer-aware expert allocation assigns more experts to deeper layers where semantic information is richer. Meanwhile, the replay strategy retains prior knowledge using minimal data, mitigating forgetting during continual training. Experiments on automatic speech recognition (ASR) and language identification (LID) demonstrate that Lamer-SSL extends self-supervised models to new languages effectively while maintaining strong performance on previously learned languages with only 2.14% parameters being trainable.