Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision
This work addresses the challenge of building performant multilingual instruction-following Speech LLMs for real-world interactive applications, offering significant gains over existing methods.
This paper tackles the problem of training multilingual instruction-following Speech LLMs with ASR-only supervision, which previously suffered from language interference. The authors introduce a language-aware distillation method that improves instruction following by 14% over multilingual baselines and achieves a 32% improvement on a new multilingual spoken QA benchmark.
Speech Large Language Models (LLMs) that understand and follow instructions in many languages are useful for real-world interaction, but are difficult to train with supervised fine-tuning, requiring large, task-specific speech corpora. While recent distillation-based approaches train performant English-only Speech LLMs using only annotated ASR data by aligning text and speech using only a lightweight projector, these models under-perform when scaled to multilingual settings due to language interference in the shared projector. We address this by introducing language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector. Our approach shows gains of 14% over matched multilingual distillation baselines on instruction following. We further synthesize Audio-MLQA, a multilingual spoken QA benchmark built on MLQA with high-quality TTS questions. Our best model improves over existing Speech LLM baselines by 32% on Audio-MLQA.