LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generation
This work addresses ASR accuracy issues for users with accents, offering a domain-specific solution that is incremental as it builds on existing LLM-based ASR methods.
The paper tackles the problem of improving automatic speech recognition (ASR) accuracy under varied acoustic conditions like accents by proposing LA-RAG, a Retrieval-Augmented Generation paradigm that uses token-level speech datastores and speech-to-speech retrieval. Experiments on Mandarin and Chinese dialect datasets show significant improvements in ASR accuracy compared to existing methods.
Recent advancements in integrating speech information into large language models (LLMs) have significantly improved automatic speech recognition (ASR) accuracy. However, existing methods often constrained by the capabilities of the speech encoders under varied acoustic conditions, such as accents. To address this, we propose LA-RAG, a novel Retrieval-Augmented Generation (RAG) paradigm for LLM-based ASR. LA-RAG leverages fine-grained token-level speech datastores and a speech-to-speech retrieval mechanism to enhance ASR accuracy via LLM in-context learning (ICL) capabilities. Experiments on Mandarin and various Chinese dialect datasets demonstrate significant improvements in ASR accuracy compared to existing methods, validating the effectiveness of our approach, especially in handling accent variations.