DreamLIP: Language-Image Pre-training with Long Captions
This work addresses the challenge of leveraging detailed image descriptions for improved multimodal learning, offering a novel approach that could benefit applications in computer vision and AI, though it is incremental in advancing existing pre-training frameworks.
The paper tackles the problem of language-image pre-training with long captions by recaptioning 30M images using a pre-trained MLLM and proposing a method to sample sub-captions for contrastive learning, resulting in a model that matches or outperforms CLIP on tasks like image-text retrieval and semantic segmentation while using fewer data (30M vs. 400M pairs).
Language-image pre-training largely relies on how precisely and thoroughly a text describes its paired image. In practice, however, the contents of an image can be so rich that well describing them requires lengthy captions (e.g., with 10 sentences), which are usually missing in existing datasets. Consequently, there are currently no clear evidences on whether and how language-image pre-training could benefit from long captions. To figure this out, we first re-caption 30M images with detailed descriptions using a pre-trained Multi-modality Large Language Model (MLLM), and then study the usage of the resulting captions under a contrastive learning framework. We observe that, each sentence within a long caption is very likely to describe the image partially (e.g., an object). Motivated by this, we propose to dynamically sample sub-captions from the text label to construct multiple positive pairs, and introduce a grouping loss to match the embeddings of each sub-caption with its corresponding local image patches in a self-supervised manner. Experimental results on a wide rage of downstream tasks demonstrate the consistent superiority of our method, termed DreamLIP, over previous alternatives, highlighting its fine-grained representational capacity. It is noteworthy that, on the tasks of image-text retrieval and semantic segmentation, our model trained with 30M image-text pairs achieves on par or even better performance than CLIP trained with 400M pairs. Project page is available at https://zyf0619sjtu.github.io/dream-lip.