LIME: Less Is More for MLLM Evaluation
This addresses the computational burden and poor discriminative power of existing MLLM benchmarks for researchers and practitioners, though it is incremental as it refines rather than revolutionizes evaluation methods.
The paper tackles the problem of inefficient and ineffective evaluation of Multimodal Large Language Models (MLLMs) by proposing LIME, a refined benchmark that reduces sample count by 76% and evaluation time by 77% while better distinguishing model capabilities.
Multimodal Large Language Models (MLLMs) are evaluated on various benchmarks, such as image captioning, visual question answering, and reasoning. However, many of these benchmarks include overly simple or uninformative samples, complicating the effective distinction of different MLLMs' performance. Furthermore, evaluating models across numerous benchmarks incurs a significant computational burden. To address these issues, we propose LIME (Less Is More for MLLM Evaluation), a refined and efficient benchmark curated through a semi-automated pipeline. This pipeline filters out uninformative samples and eliminates answer leakage by focusing on tasks that necessitate image-based understanding. Our experiments indicate that LIME reduces the number of samples by 76% and evaluation time by 77%, while also providing a more effective means of distinguishing the capabilities of different models. Notably, we find that traditional automatic metrics, such as CIDEr, are inadequate for assessing MLLMs' captioning performance; excluding the caption task score yields a more accurate reflection of overall model performance. All code and data are available at https://github.com/kangreen0210/LIME.