Are you speaking my languages? On spoken language adherence in multimodal LLMs
For developers of multilingual ASR systems, this work provides a formal definition and metric for language adherence, along with practical strategies to improve output language accuracy.
This paper addresses the problem of spoken language adherence in multimodal LLMs, where models often misidentify the output language. They propose a soft prompting approach and evaluate three mitigation strategies, finding that supervised fine-tuning and chain-of-thought reasoning reduce language violations while maintaining ASR performance.
While Large Language Model (LLM) based Automatic Speech Recognition (ASR) enables seamless multilingual use, models often misidentify the output language, compromising transcription fidelity and downstream application quality. To preserve flexibility and code-switching capabilities, we propose a soft prompting approach that hints at potential spoken languages without strictly constraining the output. We formally define this challenge as a lack of language adherence, introduce a novel metric to quantify violations, and evaluate three mitigation strategies: (1) zero-shot prompting for robust guidance under uncertainty, (2) supervised fine-tuning (SFT) to improve prompt adherence, and (3) Chain-of-Thought (CoT) reasoning to enforce adherence during decoding. We present a comparative analysis of these methods across multiple languages, evaluating effectiveness in reducing the language violation while maintaining overall ASR performance. Finally, we discuss trade-offs to guide strategy selection under various compute constraints.