WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models
This addresses the need for more efficient and accurate audio-integrated RAG in spoken dialogue models, representing a novel extension to the audio modality rather than an incremental improvement.
The paper tackles the problem of existing RAG frameworks being text-only and reliant on ASR for speech input, which loses audio information and is inefficient, by introducing WavRAG, a framework with native audio support that achieves comparable retrieval performance with a 10x speedup.
Retrieval Augmented Generation (RAG) has gained widespread adoption owing to its capacity to empower large language models (LLMs) to integrate external knowledge. However, existing RAG frameworks are primarily designed for text-based LLMs and rely on Automatic Speech Recognition to process speech input, which discards crucial audio information, risks transcription errors, and increases computational overhead. Therefore, we introduce WavRAG, the first retrieval augmented generation framework with native, end-to-end audio support. WavRAG offers two key features: 1) Bypassing ASR, WavRAG directly processes raw audio for both embedding and retrieval. 2) WavRAG integrates audio and text into a unified knowledge representation. Specifically, we propose the WavRetriever to facilitate the retrieval from a text-audio hybrid knowledge base, and further enhance the in-context capabilities of spoken dialogue models through the integration of chain-of-thought reasoning. In comparison to state-of-the-art ASR-Text RAG pipelines, WavRAG achieves comparable retrieval performance while delivering a 10x acceleration. Furthermore, WavRAG's unique text-audio hybrid retrieval capability extends the boundaries of RAG to the audio modality.