Exploring and Evaluating Multimodal Knowledge Reasoning Consistency of Multimodal Large Language Models
This work addresses a key challenge in improving MLLMs for applications requiring reliable multimodal reasoning, though it is incremental as it focuses on evaluation and analysis rather than a new solution.
The paper tackles the problem of inconsistent reasoning outcomes in multimodal large language models (MLLMs) when integrating knowledge across text and vision, by proposing four evaluation tasks and a new dataset to analyze consistency degradation, identifying contributing factors.
In recent years, multimodal large language models (MLLMs) have achieved significant breakthroughs, enhancing understanding across text and vision. However, current MLLMs still face challenges in effectively integrating knowledge across these modalities during multimodal knowledge reasoning, leading to inconsistencies in reasoning outcomes. To systematically explore this issue, we propose four evaluation tasks and construct a new dataset. We conduct a series of experiments on this dataset to analyze and compare the extent of consistency degradation in multimodal knowledge reasoning within MLLMs. Based on the experimental results, we identify factors contributing to the observed degradation in consistency. Our research provides new insights into the challenges of multimodal knowledge reasoning and offers valuable guidance for future efforts aimed at improving MLLMs.