Training Vision-Language Models with Less Bimodal Supervision
This work addresses the challenge of obtaining aligned multimodal data in low-resource settings, offering a method to reduce bimodal supervision, though it is incremental as it builds on existing models and tasks.
The study tackled the problem of reducing reliance on aligned image-text pairs for training vision-language models, finding that simpler tasks like VQAv2 and GQA can be trained without such data with minor performance loss, while for complex tasks like NLVR2, using only 5% of bimodal data or weak supervision yields moderate degradation from 74% to ~70%.
Standard practice in pretraining multimodal models, such as vision-language models, is to rely on pairs of aligned inputs from both modalities, for example, aligned image-text pairs. However, such pairs can be difficult to obtain in low-resource settings and for some modality pairs (e.g., structured tables and images). In this work, we investigate the extent to which we can reduce the reliance on such parallel data, which we term \emph{bimodal supervision}, and use models that are pretrained on each modality independently. We experiment with a high-performing vision-language model, and analyze the effect of bimodal supervision on three vision-language tasks. We find that on simpler tasks, such as VQAv2 and GQA, one can eliminate bimodal supervision completely, suffering only a minor loss in performance. Conversely, for NLVR2, which requires more complex reasoning, training without bimodal supervision leads to random performance. Nevertheless, using only 5\% of the bimodal data (142K images along with their captions), or leveraging weak supervision in the form of a list of machine-generated labels for each image, leads to only a moderate degradation compared to using 3M image-text pairs: 74\%$\rightarrow$$\sim$70\%. Our code is available at https://github.com/eladsegal/less-bimodal-sup.