CLIM: Contrastive Language-Image Mosaic for Region Representation
This addresses the challenge of expensive annotation for open-vocabulary object detection, offering a generally applicable method to enhance region representations, though it is incremental as it builds on existing detection methods.
The paper tackles the problem of learning region-text alignment for open-vocabulary object detection without costly box annotations by proposing CLIM, which uses image-text pairs to create pseudo regions in mosaicked images and trains them with contrastive loss, resulting in large-margin improvements on OV-COCO and OV-LVIS benchmarks.
Detecting objects accurately from a large or open vocabulary necessitates the vision-language alignment on region representations. However, learning such a region-text alignment by obtaining high-quality box annotations with text labels or descriptions is expensive and infeasible. In contrast, collecting image-text pairs is simpler but lacks precise object location information to associate regions with texts. In this paper, we propose a novel approach called Contrastive Language-Image Mosaic (CLIM), which leverages large-scale image-text pairs effectively for aligning region and text representations. CLIM combines multiple images into a mosaicked image and treats each image as a `pseudo region'. The feature of each pseudo region is extracted and trained to be similar to the corresponding text embedding while dissimilar from others by a contrastive loss, enabling the model to learn the region-text alignment without costly box annotations. As a generally applicable approach, CLIM consistently improves different open-vocabulary object detection methods that use caption supervision. Furthermore, CLIM can effectively enhance the region representation of vision-language models, thus providing stronger backbones for open-vocabulary object detectors. Our experimental results demonstrate that CLIM improves different baseline open-vocabulary object detectors by a large margin on both OV-COCO and OV-LVIS benchmarks. The code is available at https://github.com/wusize/CLIM.