Improved Baselines with Visual Instruction Tuning
This work makes state-of-the-art large multimodal model research more accessible by providing efficient and effective baselines, though it is incremental as it builds on existing methods like LLaVA.
The authors tackled the problem of improving large multimodal models by showing that simple modifications to LLaVA, such as using a CLIP vision encoder and adding task-specific data, establish stronger baselines that achieve state-of-the-art results across 11 benchmarks, with a 13B checkpoint trained on only 1.2M data in about one day.
Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this note, we show that the fully-connected vision-language cross-modal connector in LLaVA is surprisingly powerful and data-efficient. With simple modifications to LLaVA, namely, using CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with simple response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in ~1 day on a single 8-A100 node. We hope this can make state-of-the-art LMM research more accessible. Code and model will be publicly available.