MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspective
This addresses the need for efficient and detailed evaluation of LMMs in image generation and understanding, offering a new benchmark to guide future model optimization, though it is incremental as it builds on existing text-to-image methods.
The paper tackles the problem of evaluating Large Multimodal Models (LMMs) by proposing MMGenBench-Pipeline, a fully automated evaluation method that generates textual descriptions from images and compares them via text-to-image models, showing that many LMMs fail basic image understanding tasks despite excelling in existing benchmarks.
Large Multimodal Models (LMMs) demonstrate impressive capabilities. However, current benchmarks predominantly focus on image comprehension in specific domains, and these benchmarks are labor-intensive to construct. Moreover, their answers tend to be brief, making it difficult to assess the ability of LMMs to generate detailed descriptions of images. To address these limitations, we propose the MMGenBench-Pipeline, a straightforward and fully automated evaluation pipeline. This involves generating textual descriptions from input images, using these descriptions to create auxiliary images via text-to-image generative models, and then comparing the original and generated images. Furthermore, to ensure the effectiveness of MMGenBench-Pipeline, we design MMGenBench-Test, evaluating LMMs across 13 distinct image patterns, and MMGenBench-Domain, focusing on generative image performance. A thorough evaluation involving over 50 popular LMMs demonstrates the effectiveness and reliability of both the pipeline and benchmark. Our observations indicate that numerous LMMs excelling in existing benchmarks fail to adequately complete the basic tasks related to image understanding and description. This finding highlights the substantial potential for performance improvement in current LMMs and suggests avenues for future model optimization. Concurrently, MMGenBench-Pipeline can efficiently assess the performance of LMMs across diverse domains using only image inputs.