Multimodal LLM Augmented Reasoning for Interpretable Visual Perception Analysis
This work addresses the need for interpretable AI in Human-Computer Interaction and cognitive science, though it is incremental as it focuses on benchmarking rather than novel method development.
The paper tackles the problem of evaluating Multimodal Large Language Models (MLLMs) as cognitive assistants for visual perception tasks, proposing an annotation-free analytical framework that benchmarks MLLMs against psychological principles without developing new predictive models.
In this paper, we advance the study of AI-augmented reasoning in the context of Human-Computer Interaction (HCI), psychology and cognitive science, focusing on the critical task of visual perception. Specifically, we investigate the applicability of Multimodal Large Language Models (MLLMs) in this domain. To this end, we leverage established principles and explanations from psychology and cognitive science related to complexity in human visual perception. We use them as guiding principles for the MLLMs to compare and interprete visual content. Our study aims to benchmark MLLMs across various explainability principles relevant to visual perception. Unlike recent approaches that primarily employ advanced deep learning models to predict complexity metrics from visual content, our work does not seek to develop a mere new predictive model. Instead, we propose a novel annotation-free analytical framework to assess utility of MLLMs as cognitive assistants for HCI tasks, using visual perception as a case study. The primary goal is to pave the way for principled study in quantifying and evaluating the interpretability of MLLMs for applications in improving human reasoning capability and uncovering biases in existing perception datasets annotated by humans.