Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models
This addresses the problem of poor fine-grained visual concept recognition in LVLMs for AI researchers and developers, but it is incremental as it focuses on benchmarking rather than a novel solution.
The paper reveals that state-of-the-art Large Vision-Language Models (LVLMs) like LLaVa-1.5, InstructBLIP, and GPT-4V perform poorly in fine-grained visual categorization, with an average drop of 65.58 in EM for Stanford Dogs, and struggle to generate accurate explanations with detailed attributes. It proposes a new evaluation benchmark, Finer, to assess and improve LVLMs' fine-grained visual comprehension and explainability.
Recent advances in instruction-tuned Large Vision-Language Models (LVLMs) have imbued the models with the ability to generate high-level, image-grounded explanations with ease. While such capability is largely attributed to the rich world knowledge contained within the Large Language Models (LLMs), our work reveals their shortcomings in fine-grained visual categorization (FGVC) across six different benchmark settings. Most recent state-of-the-art LVLMs like LLaVa-1.5, InstructBLIP and GPT-4V not only severely deteriorate in terms of classification performance, e.g., average drop of 65.58 in EM for Stanford Dogs for LLaVA-1.5, but also struggle to generate an accurate explanation with detailed attributes based on the concept that appears within an input image despite their capability to generate holistic image-level descriptions. In-depth analyses show that instruction-tuned LVLMs exhibit modality gap, showing discrepancy when given textual and visual inputs that correspond to the same concept, preventing the image modality from leveraging the rich parametric knowledge within the LLMs. In an effort to further the community's endeavor in this direction, we propose a multiple granularity attribute-centric evaluation benchmark, Finer, which aims to establish a ground to evaluate LVLMs' fine-grained visual comprehension ability and provide significantly improved explainability.