Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
This work addresses the need for more detailed image comprehension in MLLMs, which is an incremental improvement for multimodal AI applications.
The paper tackles the problem of fine-grained cross-modal alignment in Multimodal Large Language Models (MLLMs) by proposing Position-enhanced Visual Instruction Tuning (PVIT), which integrates a region-level vision encoder and uses generated image-region-language data, resulting in demonstrated superiority in experiments.
Recently, Multimodal Large Language Models (MLLMs) that enable Large Language Models (LLMs) to interpret images through visual instruction tuning have achieved significant success. However, existing visual instruction tuning methods only utilize image-language instruction data to align the language and image modalities, lacking a more fine-grained cross-modal alignment. In this paper, we propose Position-enhanced Visual Instruction Tuning (PVIT), which extends the functionality of MLLMs by integrating an additional region-level vision encoder. This integration promotes a more detailed comprehension of images for the MLLM. In addition, to efficiently achieve a fine-grained alignment between the vision modules and the LLM, we design multiple data generation strategies to construct an image-region-language instruction dataset. Finally, we present both quantitative experiments and qualitative analysis that demonstrate the superiority of the proposed model. Code and data will be released at https://github.com/PVIT-official/PVIT.