To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning
This work addresses data quality issues in visual instruction tuning for multimodal AI, offering incremental improvements for researchers and developers in the field.
The paper tackles the problem of coarse-grained visual instruction data in multimodal models by introducing LVIS-Instruct4V, a fine-grained dataset generated using GPT-4V, which improves LLaVA-1.5's performance on benchmarks like LLaVA$^w$ (76.7 vs. 70.7) and MM-Vet (40.2 vs. 35.4).
Existing visual instruction tuning methods typically prompt large language models with textual descriptions to generate instruction-following data. Despite the promising performance achieved, these descriptions are derived from image annotations, which are oftentimes coarse-grained. Furthermore, the instructions might even contradict the visual content without observing the entire visual context. To address this challenge, we introduce a fine-grained visual instruction dataset, LVIS-Instruct4V, which contains 220K visually aligned and context-aware instructions produced by prompting the powerful GPT-4V with images from LVIS. Through experimental validation and case studies, we demonstrate that high-quality visual instructional data could improve the performance of LLaVA-1.5, a state-of-the-art large multimodal model, across a wide spectrum of benchmarks by clear margins. Notably, by simply replacing the LLaVA-Instruct with our LVIS-Instruct4V, we achieve better results than LLaVA on most challenging LMM benchmarks, e.g., LLaVA$^w$ (76.7 vs. 70.7) and MM-Vet (40.2 vs. 35.4). We release our data and model at https://github.com/X2FD/LVIS-INSTRUCT4V.