MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique
This work addresses the need for reliable evaluation of multimodal critique in LMMs, which is crucial for their self-improvement and use as AI assistants, though it is incremental as it builds on existing critique studies in language-only settings.
The authors tackled the problem of evaluating the critique ability of Large Multimodal Models (LMMs) by introducing MM-CRITIC, a benchmark covering 8 task types and over 500 tasks, which collected 4471 samples and used expert-informed scoring rubrics with GPT-4o for reliable assessment, revealing insights such as correlations between response quality and critique.
The ability of critique is vital for models to self-improve and serve as reliable AI assistants. While extensively studied in language-only settings, multimodal critique of Large Multimodal Models (LMMs) remains underexplored despite their growing capabilities in tasks like captioning and visual reasoning. In this work, we introduce MM-CRITIC, a holistic benchmark for evaluating the critique ability of LMMs across multiple dimensions: basic, correction, and comparison. Covering 8 main task types and over 500 tasks, MM-CRITIC collects responses from various LMMs with different model sizes and is composed of 4471 samples. To enhance the evaluation reliability, we integrate expert-informed ground answers into scoring rubrics that guide GPT-4o in annotating responses and generating reference critiques, which serve as anchors for trustworthy judgments. Extensive experiments validate the effectiveness of MM-CRITIC and provide a comprehensive assessment of leading LMMs' critique capabilities under multiple dimensions. Further analysis reveals some key insights, including the correlation between response quality and critique, and varying critique difficulty across evaluation dimensions. Our code is available at https://github.com/MichealZeng0420/MM-Critic.