Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?
This work addresses challenges in MLLMs for AI researchers, but it is incremental as it focuses on benchmarking and evaluation rather than proposing a new model or method.
The paper tackles the problem of visual fine-grained perception and commonsense causal inference in Multimodal Large Language Models (MLLMs) by introducing Argus Inspection, a benchmark with two difficulty levels, and the Eye of Panoptes framework for evaluation, finding that the highest performance in visual fine-grained reasoning is only 0.46.
As Multimodal Large Language Models (MLLMs) continue to evolve, their cognitive and reasoning capabilities have seen remarkable progress. However, challenges in visual fine-grained perception and commonsense causal inference persist. This paper introduces Argus Inspection, a multimodal benchmark with two levels of difficulty, emphasizing detailed visual recognition while incorporating real-world commonsense understanding to evaluate causal reasoning abilities. Expanding on it, we present the Eye of Panoptes framework, which integrates a binary parametric Sigmoid metric with an indicator function, enabling a more holistic evaluation of MLLMs' responses in opinion-based reasoning tasks. Experiments conducted on 26 mainstream MLLMs reveal that the highest performance in visual fine-grained reasoning reaches only 0.46, highlighting considerable potential for enhancement. Our research offers valuable perspectives for the continued refinement of MLLMs.