MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation
This work addresses the underexplored abilities of MLLMs in real-world conversational scenarios, providing a benchmark for the research community, though it is incremental as it builds on existing MLLM evaluation efforts.
The paper tackles the problem of evaluating multimodal large language models (MLLMs) in real-world conversations by introducing the MMRC benchmark, which reveals a significant accuracy drop in open-ended interactions and proposes a NOTE-TAKING strategy that improves performance across six MLLMs.
Recent multimodal large language models (MLLMs) have demonstrated significant potential in open-ended conversation, generating more accurate and personalized responses. However, their abilities to memorize, recall, and reason in sustained interactions within real-world scenarios remain underexplored. This paper introduces MMRC, a Multi-Modal Real-world Conversation benchmark for evaluating six core open-ended abilities of MLLMs: information extraction, multi-turn reasoning, information update, image management, memory recall, and answer refusal. With data collected from real-world scenarios, MMRC comprises 5,120 conversations and 28,720 corresponding manually labeled questions, posing a significant challenge to existing MLLMs. Evaluations on 20 MLLMs in MMRC indicate an accuracy drop during open-ended interactions. We identify four common failure patterns: long-term memory degradation, inadequacies in updating factual knowledge, accumulated assumption of error propagation, and reluctance to say no. To mitigate these issues, we propose a simple yet effective NOTE-TAKING strategy, which can record key information from the conversation and remind the model during its responses, enhancing conversational capabilities. Experiments across six MLLMs demonstrate significant performance improvements.