Joint Visual and Text Prompting for Improved Object-Centric Perception with Multimodal Large Language Models
This work addresses a specific challenge in VQA for MLLMs like GPT-4V and Gemini Pro, offering incremental improvements in object-oriented perception tasks.
The paper tackles the problem of improving object-centric perception in Multimodal Large Language Models (MLLMs) for Visual Question Answering (VQA) by proposing Joint Visual and Text Prompting (VTPrompt), which integrates fine-grained visual cues with textual prompts to reduce hallucinations and enhance accuracy, resulting in score improvements of up to 183.5 on MME and 8.17% to 15.69% on MMB benchmarks.
Multimodal Large Language Models (MLLMs) such as GPT-4V and Gemini Pro face challenges in achieving human-level perception in Visual Question Answering (VQA), particularly in object-oriented perception tasks which demand fine-grained understanding of object identities, locations or attributes, as indicated by empirical findings. This is mainly due to their limited capability to effectively integrate complex visual cues with textual information and potential object hallucinations. In this paper, we present a novel approach, Joint Visual and Text Prompting (VTPrompt), that employs fine-grained visual information to enhance the capability of MLLMs in VQA, especially for object-oriented perception. VTPrompt merges visual and text prompts to extract key concepts from textual questions and employs a detection model to highlight relevant objects as visual prompts in images. The processed images alongside text prompts are subsequently fed into MLLMs to produce more accurate answers. Our experiments with GPT-4V and Gemini Pro, on three benchmarks, i.e., MME , MMB and POPE, demonstrate significant improvements. Particularly, our method led to a score improvement of up to 183.5 for GPT-4V on MME and enhanced MMB performance by 8.17\% for GPT-4V and 15.69\% for Gemini Pro.