Enhancing Multi-Image Question Answering via Submodular Subset Selection
This work addresses scalability issues in multi-image question answering for users of large multimodal models, though it appears incremental as it enhances an existing retriever framework.
The paper tackles the problem of multi-image question answering where large multimodal models struggle with scalability and retrieval performance when processing many images, and demonstrates that using query-aware submodular subset selection techniques like GraphCut to pre-select relevant images improves the retriever pipeline effectiveness, particularly for large haystack sizes.
Large multimodal models (LMMs) have achieved high performance in vision-language tasks involving single image but they struggle when presented with a collection of multiple images (Multiple Image Question Answering scenario). These tasks, which involve reasoning over large number of images, present issues in scalability (with increasing number of images) and retrieval performance. In this work, we propose an enhancement for retriever framework introduced in MIRAGE model using submodular subset selection techniques. Our method leverages query-aware submodular functions, such as GraphCut, to pre-select a subset of semantically relevant images before main retrieval component. We demonstrate that using anchor-based queries and augmenting the data improves submodular-retriever pipeline effectiveness, particularly in large haystack sizes.