BRIT: Bidirectional Retrieval over Unified Image-Text Graph
This addresses the challenge of enhancing response quality in large language models for multi-modal documents, particularly when fine-tuning is ineffective, representing an incremental advancement in multi-modal RAG.
The paper tackles the problem of multi-modal retrieval-augmented generation for documents containing both text and images, proposing BRIT, a framework that unifies text-image connections into a graph to retrieve relevant content for complex cross-modal questions, with experiments showing its superiority on a new MM-RAG test set.
Retrieval-Augmented Generation (RAG) has emerged as a promising technique to enhance the quality and relevance of responses generated by large language models. While recent advancements have mainly focused on improving RAG for text-based queries, RAG on multi-modal documents containing both texts and images has not been fully explored. Especially when fine-tuning does not work. This paper proposes BRIT, a novel multi-modal RAG framework that effectively unifies various text-image connections in the document into a multi-modal graph and retrieves the texts and images as a query-specific sub-graph. By traversing both image-to-text and text-to-image paths in the graph, BRIT retrieve not only directly query-relevant images and texts but also further relevant contents to answering complex cross-modal multi-hop questions. To evaluate the effectiveness of BRIT, we introduce MM-RAG test set specifically designed for multi-modal question answering tasks that require to understand the text-image relations. Our comprehensive experiments demonstrate the superiority of BRIT, highlighting its ability to handle cross-modal questions on the multi-modal documents.