PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMs
This work addresses the problem of multilingual speech-to-text alignment for SLMs, offering a novel method to enhance performance in this domain.
The paper tackled the challenge of aligning speech and text representations in multilingual Speech Large Models (SLMs) by introducing Progressive Alignment Representation Training (PART), a multi-stage and multi-task framework that separates within-language from cross-language alignment, resulting in improved performance over conventional approaches on datasets like CommonVoice 15, Fleurs, Wenetspeech, and CoVoST2.
Large language models (LLMs) have expanded from text to speech, giving rise to Speech Large Models (SLMs) that support recognition, translation, and synthesis. A key challenge is aligning speech and text representations, which becomes harder in multilingual settings. Existing methods often freeze LLM parameters and train encoders on multilingual data, but this forces cross-language convergence and limits performance. We introduce Progressive Alignment Representation Training (PART), a multi-stage and multi-task framework that separates within-language from cross-language alignment. During cross-language training, LLM parameters are dynamically activated, and text-based tasks are later introduced to enhance multilingual understanding. Experiments on CommonVoice 15, Fleurs, Wenetspeech, and CoVoST2 show that PART surpasses conventional approaches, with analysis confirming its ability to balance language-specific distinctions and cross-language generalization. These results demonstrate PART's effectiveness and generality for multilingual speech modality alignment.