MMGist: A Comprehensive Multimodal Benchmark for 2027
For researchers evaluating large vision-language models, this work provides a more efficient and discriminative benchmark that addresses key flaws in current evaluation practices.
The authors identify three major issues in existing vision-language benchmarks—lack of visual dependency, performance saturation, and anomalous items—and propose MMGist, a curated benchmark of 7,262 items. MMGist reduces evaluation items by 69% while improving cross-model discrimination by 78% and preserving model rankings with Spearman ρ=0.98.
We conduct a systematic study of 18 widely used vision-language benchmarks and identify three major issues: 1) many items do not rely on visual cues and therefore fail to effectively measure multimodal understanding; 2) many items are already close to performance saturation for current LVLMs, which limits their discriminative power; 3) a small number of anomalous items affect the reliability of evaluation results. To this end, we propose MMGist, a curated benchmark that covers seven capability dimensions and contains 7,262 items. MMGist is constructed through a three-stage pipeline, which sequentially combines text-ablation filtering, cross-model saturation filtering, and anomaly detection filtering. We conduct extensive experiments on 27 leading LVLMs and compare MMGist with the raw pool of 23,250 items. The results show that MMGist preserves model rankings with high fidelity, with Spearman $ρ= 0.98$, while reducing evaluation items by 69\% and improving cross-model discrimination by 78\%. Further results indicate that Visual Logic remains a systematic weakness of current LVLMs, while knowledge-intensive dimensions such as Expert Knowledge dimensions remain important factors for distinguishing closed-source models from open-source models. These findings suggest that high-quality evaluation should prioritize visual dependency, discriminative power, and reliability, rather than simply pursuing benchmark scale.