SSR: Alignment-Aware Modality Connector for Speech Language Models
This work provides a more efficient and effective way to integrate speech into large language models, which is significant for researchers and developers working on multimodal AI systems, particularly those dealing with long-form speech inputs.
This paper addresses the challenges of integrating long-form speech into pre-trained language models, specifically inefficient encoding and catastrophic forgetting of text modality. The proposed SSR-Connector segments and compresses speech features using speech-text alignments, and a two-stage training pipeline (distillation and fine-tuning) is introduced to mitigate forgetting. The method achieves significant improvements in speech understanding, with +10 accuracy on StoryCloze and +20 on Speech-MMLU, while maintaining text abilities.
Fusing speech into pre-trained language model (SpeechLM) usually suffers from inefficient encoding of long-form speech and catastrophic forgetting of pre-trained text modality. We propose SSR-Connector (Segmented Speech Representation Connector) for better modality fusion. Leveraging speech-text alignments, our approach segments and compresses speech features to match the granularity of text embeddings. Additionally, we introduce a two-stage training pipeline that includes the distillation and fine-tuning phases to mitigate catastrophic forgetting. SSR-Connector outperforms existing mechanism for speech-text modality fusion, consistently achieving better speech understanding (e.g., +10 accuracy on StoryCloze and +20 on Speech-MMLU) while preserving pre-trained text ability.