ATIR: Towards Audio-Text Interleaved Contextual Retrieval
This addresses the gap in multimodal retrieval research, which has overlooked audio, for applications requiring low-latency processing and richer audio context.
The paper tackles the problem of multimodal information retrieval by introducing the Audio-Text Interleaved contextual Retrieval (ATIR) task, where queries alternate between audio and text, and constructs a benchmark that unifies four types of contextual retrieval tasks, with their ATIR model showing substantial improvements over baselines.
Audio carries richer information than text, including emotion, speaker traits, and environmental context, while also enabling lower-latency processing compared to speech-to-text pipelines. However, recent multimodal information retrieval research has predominantly focused on images, largely overlooking audio, especially in the setting of interleaved audio-text contextual retrieval. In this work, we introduce the Audio-Text Interleaved contextual Retrieval (ATIR) task, where queries can alternate between audio and text modalities. We construct an ATIR benchmark by integrating several Automatic Speech Recognition (ASR), QA, and retrieval datasets, ultimately unifying four types of contextual retrieval tasks. This benchmark substantially addresses the limitations of existing audio retrieval datasets in semantic retrieval. To study this task, we evaluate several off-the-shelf retrievers and train our ATIR model based on a Multimodal Large Language Model (MLLM). We further introduce a novel token compression mechanism that is orthogonal to existing compression methods, thereby alleviating the issue of excessive audio tokens in MLLM-based ATIR models. Experimental results demonstrate that our ATIR model achieves substantial improvements over strong baselines.