Enhancing Speech Large Language Models through Reinforced Behavior Alignment
This work addresses the challenge of improving speech-based AI models for better user interaction, representing an incremental advancement in multimodal language processing.
The paper tackles the performance gap in speech-based large language models (SpeechLMs) compared to text-based models in instruction-following by introducing a Reinforced Behavior Alignment (RBA) framework, which uses self-synthesized data and reinforcement learning to enhance capabilities, achieving state-of-the-art results on tasks like spoken question answering and speech-to-text translation.
The recent advancements of Large Language Models (LLMs) have spurred considerable research interest in extending their linguistic capabilities beyond text to other modalities, which leads to emergence of speech-based LLMs (SpeechLMs) with capability of processing user request in either speech or textual formats. However, owing to inter-modal discrepancies, these SpeechLMs still exhibit a significant performance gap compared to their text-based LLM counterparts in instruction-following, particularly when confronted with the dynamic and variable nature of user speech. To address this challenge, this paper introduces a framework termed Reinforced Behavior Alignment (RBA), designed to bolster the language generation proficiency of SpeechLMs. Instead of relying on supervised fine-tuning from human annotations, RBA employs a self-synthesis methodology to generate extensive, high-fidelity alignment data by a powerful teacher LLM. Then SpeechLMs is aligned its behavior with that of a teacher using a reinforcement learning-based approach. Experimental results demonstrate that this method effectively enhances the instruction-following capabilities of SpeechLMs that outperform conventional distillation baselines. Crucially, we demonstrate that RBA can be seamlessly extended to tasks such including spoken question answering and speech-to-text translation, attaining state-of-the-art performance on open benchmarks with only self-generated data.