Hierarchical Scheduling for Multi-Vector Image Retrieval
This work addresses efficiency and accuracy issues in image retrieval for RAG-based MLLM applications, representing an incremental improvement over prior multi-vector retrieval methods.
The paper tackles the problem of sub-optimal accuracy and efficiency in multi-vector image retrieval for multimodal large language models by proposing HiMIR, a hierarchical scheduling framework that enhances alignment with image objects and reduces redundancy, resulting in substantial accuracy improvements and up to 3.5 times computation reduction compared to existing systems.
To effectively leverage user-specific data, retrieval augmented generation (RAG) is employed in multimodal large language model (MLLM) applications. However, conventional retrieval approaches often suffer from limited retrieval accuracy. Recent advances in multi-vector retrieval (MVR) improve accuracy by decomposing queries and matching against segmented images. They still suffer from sub-optimal accuracy and efficiency, overlooking alignment between the query and varying image objects and redundant fine-grained image segments. In this work, we present an efficient scheduling framework for image retrieval - HiMIR. First, we introduce a novel hierarchical paradigm, employing multiple intermediate granularities for varying image objects to enhance alignment. Second, we minimize redundancy in retrieval by leveraging cross-hierarchy similarity consistency and hierarchy sparsity to minimize unnecessary matching computation. Furthermore, we configure parameters for each dataset automatically for practicality across diverse scenarios. Our empirical study shows that, HiMIR not only achieves substantial accuracy improvements but also reduces computation by up to 3.5 times over the existing MVR system.