Efficient Multimodal Planning Agent for Visual Question-Answering
This work addresses efficiency bottlenecks in VQA systems for AI applications, representing an incremental improvement over existing mRAG methods.
The paper tackles the inefficiency of multi-stage multimodal Retrieval-Augmented Generation (mRAG) pipelines in Visual Question-Answering (VQA) by proposing a multimodal planning agent that dynamically decomposes the pipeline to reduce redundant computations, resulting in a 60% reduction in search time while outperforming baselines across six datasets.
Visual Question-Answering (VQA) is a challenging multimodal task that requires integrating visual and textual information to generate accurate responses. While multimodal Retrieval-Augmented Generation (mRAG) has shown promise in enhancing VQA systems by providing more evidence on both image and text sides, the default procedure that addresses VQA queries, especially the knowledge-intensive ones, often relies on multi-stage pipelines of mRAG with inherent dependencies. To mitigate the inefficiency limitations while maintaining VQA task performance, this paper proposes a method that trains a multimodal planning agent, dynamically decomposing the mRAG pipeline to solve the VQA task. Our method optimizes the trade-off between efficiency and effectiveness by training the agent to intelligently determine the necessity of each mRAG step. In our experiments, the agent can help reduce redundant computations, cutting search time by over 60\% compared to existing methods and decreasing costly tool calls. Meanwhile, experiments demonstrate that our method outperforms all baselines, including a Deep Research agent and a carefully designed prompt-based method, on average over six various datasets. Code will be released.