A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions
This work addresses the need for better benchmarks in vision-language modeling by providing a dense captioning dataset, though it is incremental as it builds on existing CLIP-style models.
The authors tackled the problem of evaluating vision-language models' understanding of detailed image content by collecting the Densely Captioned Images (DCI) dataset with 7805 images and captions averaging over 1000 words each, and showed that finetuning CLIP on a summarized version (sDCI) led to significant improvements over the baseline.
Curation methods for massive vision-language datasets trade off between dataset size and quality. However, even the highest quality of available curated captions are far too short to capture the rich visual detail in an image. To show the value of dense and highly-aligned image-text pairs, we collect the Densely Captioned Images (DCI) dataset, containing 7805 natural images human-annotated with mask-aligned descriptions averaging above 1000 words each. With precise and reliable captions associated with specific parts of an image, we can evaluate vision-language models' (VLMs) understanding of image content with a novel task that matches each caption with its corresponding subcrop. As current models are often limited to 77 text tokens, we also introduce a summarized version (sDCI) in which each caption length is limited. We show that modern techniques that make progress on standard benchmarks do not correspond with significant improvement on our sDCI based benchmark. Lastly, we finetune CLIP using sDCI and show significant improvements over the baseline despite a small training set. By releasing the first human annotated dense image captioning dataset, we hope to enable the development of new benchmarks or fine-tuning recipes for the next generation of VLMs to come.