DetailCLIP: Injecting Image Details into CLIP's Feature Space
This work addresses a limitation in CLIP-like models for computer vision applications, offering an incremental improvement for tasks requiring fine-grained detail retrieval.
The paper tackles the problem of CLIP losing subtle details in high-resolution images due to its fixed input size, by introducing an efficient framework that injects image details into CLIP's feature space, resulting in significant performance improvements in image retrieval tasks on real-world and synthetic datasets.
Although CLIP-like Visual Language Models provide a functional joint feature space for image and text, due to the limitation of the CILP-like model's image input size (e.g., 224), subtle details are lost in the feature representation if we input high-resolution images (e.g., 2240). In this work, we introduce an efficient framework that can produce a single feature representation for a high-resolution image that injects image details and shares the same semantic space as the original CLIP. In the framework, we train a feature fusing model based on CLIP features extracted from a carefully designed image patch method that can cover objects of any scale, weakly supervised by image-agnostic class prompted queries. We validate our framework by retrieving images from class prompted queries on the real world and synthetic datasets, showing significant performance improvement on these tasks. Furthermore, to fully demonstrate our framework's detail retrieval ability, we construct a CLEVR-like synthetic dataset called CLVER-DS, which is fully annotated and has a controllable object scale.