Visual Question Decomposition on Multimodal Large Language Models
This addresses the problem of enhancing complex question-answering in multimodal AI systems, representing an incremental advancement by adapting existing decomposition strategies to MLLMs.
The paper tackles the unexplored capability of Multimodal Large Language Models (MLLMs) in visual question decomposition, revealing their struggles and proposing a finetuning dataset and pipeline that significantly improve sub-question quality and achieve higher accuracy on VQA benchmarks.
Question decomposition has emerged as an effective strategy for prompting Large Language Models (LLMs) to answer complex questions. However, while existing methods primarily focus on unimodal language models, the question decomposition capability of Multimodal Large Language Models (MLLMs) has yet to be explored. To this end, this paper explores visual question decomposition on MLLMs. Specifically, we introduce a systematic evaluation framework including a dataset and several evaluation criteria to assess the quality of the decomposed sub-questions, revealing that existing MLLMs struggle to produce high-quality sub-questions. To address this limitation, we propose a specific finetuning dataset, DecoVQA+, for enhancing the model's question decomposition capability. Aiming at enabling models to perform appropriate selective decomposition, we propose an efficient finetuning pipeline. The finetuning pipeline consists of our proposed dataset and a training objective for selective decomposition. Finetuned MLLMs demonstrate significant improvements in the quality of sub-questions and the policy of selective question decomposition. Additionally, the models also achieve higher accuracy with selective decomposition on VQA benchmark datasets.