VP-Bench: A Comprehensive Benchmark for Visual Prompting in Multimodal Large Language Models
This work addresses a gap in benchmarking for MLLMs, providing a tool for researchers and developers to evaluate and improve VP capabilities, though it is incremental as it focuses on assessment rather than novel model development.
The paper tackles the lack of systematic evaluation for visual prompting (VP) in multimodal large language models (MLLMs) by introducing VP-Bench, a benchmark that assesses VP perception and utilization, and finds that current models vary in their ability to interpret VPs, with evaluations on 28 MLLMs including GPT-4o and InternVL3.
Multimodal large language models (MLLMs) have enabled a wide range of advanced vision-language applications, including fine-grained object recognition and contextual understanding. When querying specific regions or objects in an image, human users naturally use "visual prompts" (VPs), such as bounding boxes, to provide reference. However, no existing benchmark systematically evaluates the ability of MLLMs to interpret such VPs. This gap leaves it unclear whether current MLLMs can effectively recognize VPs, an intuitive prompting method for humans, and use them to solve problems. To address this limitation, we introduce VP-Bench, a benchmark for assessing MLLMs' capability in VP perception and utilization. VP-Bench employs a two-stage evaluation framework: Stage 1 examines models' ability to perceive VPs in natural scenes, using 30k visualized prompts spanning eight shapes and 355 attribute combinations. Stage 2 investigates the impact of VPs on downstream tasks, measuring their effectiveness in real-world problem-solving scenarios. Using VP-Bench, we evaluate 28 MLLMs, including proprietary systems (e.g., GPT-4o) and open-source models (e.g., InternVL3 and Qwen2.5-VL), and provide a comprehensive analysis of factors that affect VP understanding, such as variations in VP attributes, question arrangement, and model scale. VP-Bench establishes a new reference framework for studying how MLLMs comprehend and resolve grounded referring questions.