AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning
This work addresses the problem of poor generalization and high data curation costs in speech-LLMs for researchers and practitioners, offering a more efficient and scalable approach.
The paper tackles the challenge of extending large language models to speech without requiring task-specific instruction-tuning data by proposing Self-Generated Instruction-Free Tuning (SIFT), which uses a frozen LLM to generate supervision from speech-text pairs. The resulting model, AZeroS, achieves state-of-the-art performance on semantic and paralinguistic benchmarks like VoiceBench and AIR-Bench, despite training on only about 28,000 hours of data and updating minimal parameters.
Extending large language models (LLMs) to the speech domain has recently gained significant attention. A typical approach connects a pretrained LLM with an audio encoder through a projection module and trains the resulting model on large-scale, task-specific instruction-tuning datasets. However, curating such instruction-tuning data for specific requirements is time-consuming, and models trained in this manner often generalize poorly to unseen tasks. In this work, we first formulate that the strongest generalization of a speech-LLM is achieved when it is trained with Self-Generated Instruction-Free Tuning (SIFT), in which supervision signals are generated by a frozen LLM using textual representations of speech as input. Our proposed SIFT paradigm eliminates the need for collecting task-specific question-answer pairs and yields the theoretically best generalization to unseen tasks. Building upon this paradigm, we introduce AZeroS (Auden Zero-instruction-tuned Speech-LLM), which is trained on speech-text pairs derived from publicly available corpora, including approximately 25,000 hours of speech with ASR transcripts and 3,000 hours of speech with paralinguistic labels. Built upon Qwen2.5-7B-Instruct, the model updates only two lightweight projection modules (23.8 million parameters each), while keeping both the LLM and audio encoders frozen. Despite the minimal training cost and modest data scale, AZeroS achieves state-of-the-art performance on both semantic and paralinguistic benchmarks, including VoiceBench, AIR-Bench Foundation (Speech), and AIR-Bench Chat (Speech).