Rethinking Overlooked Aspects in Vision-Language Models
This work addresses data optimization for researchers developing vision-language models, but it is incremental as it builds on existing modular architectures like LLaVA without introducing a new model.
The paper tackled the problem of data inefficiency in vision-language models by showing that increasing pre-training data can degrade performance and that not all instruction tuning data is necessary, establishing a pipeline to identify the most efficient datasets.
Recent advancements in large vision-language models (LVLMs), such as GPT4-V and LLaVA, have been substantial. LLaVA's modular architecture, in particular, offers a blend of simplicity and efficiency. Recent works mainly focus on introducing more pre-training and instruction tuning data to improve model's performance. This paper delves into the often-neglected aspects of data efficiency during pre-training and the selection process for instruction tuning datasets. Our research indicates that merely increasing the size of pre-training data does not guarantee improved performance and may, in fact, lead to its degradation. Furthermore, we have established a pipeline to pinpoint the most efficient instruction tuning (SFT) dataset, implying that not all SFT data utilized in existing studies are necessary. The primary objective of this paper is not to introduce a state-of-the-art model, but rather to serve as a roadmap for future research, aiming to optimize data usage during pre-training and fine-tuning processes to enhance the performance of vision-language models.