Visual Prompting in Multimodal Large Language Models: A Survey
It addresses the need for a systematic review of visual prompting in MLLMs for researchers and practitioners, but it is incremental as it surveys existing methods rather than introducing new ones.
This survey paper tackles the problem of understanding and categorizing visual prompting methods in multimodal large language models (MLLMs), providing a comprehensive overview of techniques for visual prompting, prompt generation, compositional reasoning, and prompt learning to enhance visual capabilities.
Multimodal large language models (MLLMs) equip pre-trained large-language models (LLMs) with visual capabilities. While textual prompting in LLMs has been widely studied, visual prompting has emerged for more fine-grained and free-form visual instructions. This paper presents the first comprehensive survey on visual prompting methods in MLLMs, focusing on visual prompting, prompt generation, compositional reasoning, and prompt learning. We categorize existing visual prompts and discuss generative methods for automatic prompt annotations on the images. We also examine visual prompting methods that enable better alignment between visual encoders and backbone LLMs, concerning MLLM's visual grounding, object referring, and compositional reasoning abilities. In addition, we provide a summary of model training and in-context learning methods to improve MLLM's perception and understanding of visual prompts. This paper examines visual prompting methods developed in MLLMs and provides a vision of the future of these methods.