BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models
This work addresses the need for more robust LMMs in practical applications by introducing a benchmark to assess cross-style capabilities, though it is incremental as it builds on existing models without new architectural innovations.
The paper tackles the problem of evaluating the robustness of Large Multimodal Models (LMMs) against diverse image style shifts, revealing that LMMs suffer performance degradation with styles like artistic, sensor, and application variations, and proposes a training-free prompting method to enhance their reasoning.
Large Multimodal Models (LMMs) such as GPT-4V and LLaVA have shown remarkable capabilities in visual reasoning with common image styles. However, their robustness against diverse style shifts, crucial for practical applications, remains largely unexplored. In this paper, we propose a new benchmark, BenchLMM, to assess the robustness of LMMs against three different styles: artistic image style, imaging sensor style, and application style, where each style has five sub-styles. Utilizing BenchLMM, we comprehensively evaluate state-of-the-art LMMs and reveal: 1) LMMs generally suffer performance degradation when working with other styles; 2) An LMM performs better than another model in common style does not guarantee its superior performance in other styles; 3) LMMs' reasoning capability can be enhanced by prompting LMMs to predict the style first, based on which we propose a versatile and training-free method for improving LMMs; 4) An intelligent LMM is expected to interpret the causes of its errors when facing stylistic variations. We hope that our benchmark and analysis can shed new light on developing more intelligent and versatile LMMs.