Elevating Visual Question Answering through Implicitly Learned Reasoning Pathways in LVLMs
This addresses the limitation of LVLMs in multi-step visual reasoning for tasks like visual question answering, representing an incremental advancement over existing methods.
The paper tackles the problem of complex visual reasoning in Large Vision-Language Models (LVLMs) by proposing MF-SQ-LLaVA, which enhances performance through implicit self-questioning and multi-task training, achieving significant improvements over state-of-the-art models on ScienceQA and VQAv2 datasets.
Large Vision-Language Models (LVLMs) have shown remarkable progress in various multimodal tasks, yet they often struggle with complex visual reasoning that requires multi-step inference. To address this limitation, we propose MF-SQ-LLaVA, a novel approach that enhances LVLMs by enabling implicit self-questioning through end-to-end training. Our method involves augmenting visual question answering datasets with reasoning chains consisting of sub-question and answer pairs, and training the LVLM with a multi-task loss that encourages the generation and answering of these intermediate steps, as well as the prediction of the final answer. We conduct extensive experiments on the ScienceQA and VQAv2 datasets, demonstrating that MF-SQ-LLaVA significantly outperforms existing state-of-the-art models, including the base LLaVA and the original SQ-LLaVA. Ablation studies further validate the contribution of each component of our approach, and human evaluation confirms the improved accuracy and coherence of the reasoning process enabled by our method.