Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment
This work addresses the problem of high computational costs in vision-language pretraining, making it more accessible for academic labs with limited resources, though it is incremental in improving efficiency.
The paper tackles the computational inefficiency of large-scale vision-language pretraining by proposing ViCHA, a framework that uses hierarchical alignment, masked image modeling, and visual concepts to achieve better performance on tasks like Image-Text Retrieval and VQA while using four times less data.
Vision and Language Pretraining has become the prevalent approach for tackling multimodal downstream tasks. The current trend is to move towards ever larger models and pretraining datasets. This computational headlong rush does not seem reasonable in the long term to move toward sustainable solutions, and de facto excludes academic laboratories with limited resources. In this work, we propose a new framework, dubbed ViCHA, that efficiently exploits the input data to boost the learning by: (a) a new hierarchical cross-modal alignment loss, (b) new self-supervised scheme based on masked image modeling, (c) leveraging image-level annotations, called Visual Concepts, obtained with existing foundation models such as CLIP to boost the performance of the image encoder. Although pretrained on four times less data, our ViCHA strategy outperforms other approaches on several downstream tasks such as Image-Text Retrieval, VQA, Visual Reasoning, Visual Entailment and Visual Grounding. The code will be made publicly available here: https://github.com/mshukor/ViCHA