Improving Contextual ASR via Multi-grained Fusion with Large Language Models
This addresses the challenge of accurately recognizing contextually relevant keywords like proper nouns in ASR, which is important for applications requiring precise transcription, though it appears incremental as it builds on existing fusion methods.
The paper tackles the problem of improving keyword recognition in Automatic Speech Recognition (ASR) by proposing a multi-grained fusion approach that combines token-level and phrase-level fusion with Large Language Models (LLMs), achieving state-of-the-art performance on keyword-related metrics while maintaining high accuracy on non-keyword text.
While end-to-end Automatic Speech Recognition (ASR) models have shown impressive performance in transcribing general speech, they often struggle to accurately recognize contextually relevant keywords, such as proper nouns or user-specific entities. Previous approaches have explored leveraging keyword dictionaries in the textual modality to improve keyword recognition, either through token-level fusion that guides token-by-token generation or phrase-level fusion that enables direct copying of keyword phrases. However, these methods operate at different granularities and have their own limitations. In this paper, we propose a novel multi-grained fusion approach that jointly leverages the strengths of both token-level and phrase-level fusion with Large Language Models (LLMs). Our approach incorporates a late-fusion strategy that elegantly combines ASR's acoustic information with LLM's rich contextual knowledge, balancing fine-grained token precision with holistic phrase-level understanding. Experiments on Chinese and English datasets demonstrate that our approach achieves state-of-the-art performance on keyword-related metrics while preserving high accuracy on non-keyword text. Ablation studies further confirm that the token-level and phrase-level components both contribute significantly to the performance gains, complementing each other in our joint multi-grained framework. The code and models will be publicly available at https://github.com/.