Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectories
This addresses the underexplored challenge of data selection for LVLMs, offering a principled solution to reduce training costs and redundancy for researchers and practitioners in vision-language AI.
The paper tackles the problem of data-efficient instruction tuning for Large Vision-Language Models (LVLMs) by proposing XMAS, a method that clusters examples based on cross-modal attention trajectories to remove redundancy; it achieves performance preservation while discarding 50% of the LLaVA-665k dataset and 85% of the Vision-Flan dataset, speeding up training by 1.2x and outperforming baselines by 30% in data reduction.
Data-efficient learning aims to eliminate redundancy in large training datasets by training models on smaller subsets of the most informative examples. While data selection has been extensively explored for vision models and large language models (LLMs), it remains underexplored for Large Vision-Language Models (LVLMs). Notably, none of existing methods can outperform random selection at different subset sizes. In this work, we propose the first principled method for data-efficient instruction tuning of LVLMs. We prove that examples with similar cross-modal attention matrices during instruction tuning have similar gradients. Thus, they influence model parameters in a similar manner and convey the same information to the model during training. Building on this insight, we propose XMAS, which clusters examples based on the trajectories of the top singular values of their attention matrices obtained from fine-tuning a small proxy LVLM. By sampling a balanced subset from these clusters, XMAS effectively removes redundancy in large-scale LVLM training data. Extensive experiments show that XMAS can discard 50% of the LLaVA-665k dataset and 85% of the Vision-Flan dataset while fully preserving performance of LLaVA-1.5-7B on 10 downstream benchmarks and speeding up its training by 1.2x. This is 30% more data reduction compared to the best baseline for LLaVA-665k. The project's website can be found at https://bigml-cs-ucla.github.io/XMAS-project-page/.