Understanding Textual Capability Degradation in Speech LLMs via Parameter Importance Analysis
This addresses a critical issue for developers and users of speech-enabled LLMs, where integrating speech weakens core textual abilities, but the work is incremental as it builds on existing fine-tuning and adaptation methods.
The paper tackles the problem of textual capability degradation in speech-enabled Large Language Models (LLMs) by analyzing the encoder-adaptor paradigm, revealing that fine-tuning disrupts parameter importance distribution for textual reasoning. It shows that mitigation strategies like layer-wise learning rate scheduling and LoRA better maintain textual competence and improve spoken question answering performance compared to full fine-tuning.
The integration of speech into Large Language Models (LLMs) has substantially expanded their capabilities, but often at the cost of weakening their core textual competence. This degradation limits the ability of speech-enabled LLMs to fully exploit their pre-trained text-based knowledge. In this work, we analyze the underlying mechanisms of this issue through a focused study of the widely used encoder-adaptor paradigm. We propose an analytical framework based on parameter importance estimation, which reveals that fine-tuning for speech introduces a textual importance distribution shift: the layer-wise allocation of parameters critical to textual reasoning is disrupted. Building on this insight, we investigate two mitigation strategies: layer-wise learning rate scheduling and Low-Rank Adaptation (LoRA), both aim to preserve the original parameter distribution. Experimental results show that both approaches better maintain textual competence than full fine-tuning, while also improving downstream spoken question answering performance. Furthermore, our analysis offers a principled explanation for the effectiveness of the proposed mitigation strategies, linking their benefits to the structural properties of textual knowledge in LLMs.